2023-11-30T21:15:00.430 INFO:root:teuthology version: 0.0.1.dev212+g595fad1 2023-11-30T21:15:00.430 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --verbose --name yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472522 -- /home/teuthworker/archive/yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472522/orig.config.yaml 2023-11-30T21:15:00.480 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-30T21:15:00.557 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472522 branch: wip-yuri8-testing-2023-11-29-0706-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7472522' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri8-testing-2023-11-29-0706-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2023-11-29-0706-pacific sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 8528 sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri8-testing-2023-11-29-0706-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKKRK2atzkSY9j+qZTWr7qs/a/nYUKdIusUMpUbHzkWBMMbs9R5dv2bBqbn2OrOcr9rcbdFU5OI3O/Zr3l7/jTs= smithi049.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDxM9L1/quSHWDQf/7twLLjDuknYBeaRc7dwxOgYHbExgrqE1RlX6dkiwi3ClkqBEa8Gve7LPj3Gv5Sv63MgfpA= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: wip-package-queries teuthology_sha1: 595fad1080bd379e3c0231a8e02595924c59a9da timestamp: 2023-11-30_16:00:27 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-11-30T21:15:00.558 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa; will attempt to use it 2023-11-30T21:15:00.558 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks 2023-11-30T21:15:00.558 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-30T21:15:00.560 INFO:teuthology.task.internal:Checking packages... 2023-11-30T21:15:00.583 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9' 2023-11-30T21:15:00.583 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-30T21:15:00.583 INFO:teuthology.packaging:ref: None 2023-11-30T21:15:00.583 INFO:teuthology.packaging:tag: None 2023-11-30T21:15:00.583 INFO:teuthology.packaging:branch: wip-yuri8-testing-2023-11-29-0706-pacific 2023-11-30T21:15:00.583 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-11-30T21:15:00.583 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri8-testing-2023-11-29-0706-pacific 2023-11-30T21:15:00.815 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-328.g0dc074ce 2023-11-30T21:15:00.815 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-30T21:15:00.822 INFO:teuthology.task.internal:no buildpackages task found 2023-11-30T21:15:00.822 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-30T21:15:00.832 INFO:teuthology.task.internal:Saving configuration 2023-11-30T21:15:00.851 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-30T21:15:00.857 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-30T21:15:00.885 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472522', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-30 21:08:06.758714', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKKRK2atzkSY9j+qZTWr7qs/a/nYUKdIusUMpUbHzkWBMMbs9R5dv2bBqbn2OrOcr9rcbdFU5OI3O/Zr3l7/jTs='} 2023-11-30T21:15:00.906 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472522', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-30 21:08:06.760380', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDxM9L1/quSHWDQf/7twLLjDuknYBeaRc7dwxOgYHbExgrqE1RlX6dkiwi3ClkqBEa8Gve7LPj3Gv5Sv63MgfpA='} 2023-11-30T21:15:00.906 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-30T21:15:00.912 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-11-30T21:15:00.912 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-11-30T21:15:00.912 INFO:teuthology.run_tasks:Running task console_log... 2023-11-30T21:15:00.984 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fced753ea60>, signals=[15]) 2023-11-30T21:15:00.984 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-30T21:15:00.989 INFO:teuthology.task.internal:Opening connections... 2023-11-30T21:15:00.990 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2023-11-30T21:15:00.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:15:01.062 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2023-11-30T21:15:01.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:15:01.132 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-30T21:15:01.138 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2023-11-30T21:15:01.162 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2023-11-30T21:15:01.165 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2023-11-30T21:15:01.225 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2023-11-30T21:15:01.225 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="8" 2023-11-30T21:15:01.225 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2023-11-30T21:15:01.225 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2023-11-30T21:15:01.225 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="8" 2023-11-30T21:15:01.225 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el8" 2023-11-30T21:15:01.225 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-30T21:15:01.225 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2023-11-30T21:15:01.225 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-30T21:15:01.225 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2023-11-30T21:15:01.225 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-30T21:15:01.226 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-30T21:15:01.226 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-30T21:15:01.226 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2023-11-30T21:15:01.247 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2023-11-30T21:15:01.269 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2023-11-30T21:15:01.270 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2023-11-30T21:15:01.332 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream" 2023-11-30T21:15:01.332 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="8" 2023-11-30T21:15:01.332 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2023-11-30T21:15:01.332 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2023-11-30T21:15:01.332 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="8" 2023-11-30T21:15:01.333 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el8" 2023-11-30T21:15:01.333 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-30T21:15:01.333 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2023-11-30T21:15:01.333 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-30T21:15:01.333 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/" 2023-11-30T21:15:01.333 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-30T21:15:01.333 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-30T21:15:01.333 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-30T21:15:01.334 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2023-11-30T21:15:01.354 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-30T21:15:01.363 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-30T21:15:01.369 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-30T21:15:01.369 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2023-11-30T21:15:01.372 DEBUG:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2023-11-30T21:15:01.392 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-30T21:15:01.398 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-30T21:15:01.398 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2023-11-30T21:15:01.433 DEBUG:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2023-11-30T21:15:01.464 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-30T21:15:01.520 INFO:teuthology.run_tasks:Running task kernel... 2023-11-30T21:15:01.541 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-30T21:15:01.542 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-30T21:15:01.542 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-30T21:15:01.543 DEBUG:teuthology.orchestra.run.smithi049:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-30T21:15:01.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-30T21:15:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-30T21:15:01.564 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2023-11-30T21:15:01.565 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2023-11-30T21:15:01.627 INFO:teuthology.orchestra.run.smithi049.stdout:4.18.0-489.el8.x86_64 2023-11-30T21:15:01.627 INFO:teuthology.task.kernel:Running kernel on smithi049: 4.18.0-489.el8.x86_64 2023-11-30T21:15:01.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum install -y kernel 2023-11-30T21:15:01.628 INFO:teuthology.orchestra.run.smithi027.stdout:4.18.0-489.el8.x86_64 2023-11-30T21:15:01.628 INFO:teuthology.task.kernel:Running kernel on smithi027: 4.18.0-489.el8.x86_64 2023-11-30T21:15:01.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2023-11-30T21:15:07.219 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - AppStream 11 MB/s | 34 MB 00:03 2023-11-30T21:15:07.302 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - AppStream 12 MB/s | 34 MB 00:02 2023-11-30T21:15:19.014 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 55 MB 00:04 2023-11-30T21:15:26.654 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - BaseOS 4.3 MB/s | 55 MB 00:12 2023-11-30T21:15:27.755 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - Extras 60 kB/s | 18 kB 00:00 2023-11-30T21:15:28.364 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-11-30T21:15:28.699 INFO:teuthology.orchestra.run.smithi049.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-11-30T21:15:35.362 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - Extras 49 kB/s | 18 kB 00:00 2023-11-30T21:15:35.904 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2023-11-30T21:15:36.239 INFO:teuthology.orchestra.run.smithi027.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-11-30T21:15:38.650 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 16 MB 00:02 2023-11-30T21:15:42.715 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 508 kB/s | 24 kB 00:00 2023-11-30T21:15:46.506 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-30T21:15:46.507 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-30T21:15:46.812 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:15:46.813 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:15:46.813 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2023-11-30T21:15:46.813 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:15:46.813 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:15:46.813 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-11-30T21:15:46.813 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-11-30T21:15:46.814 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-11-30T21:15:46.814 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-11-30T21:15:46.814 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:15:46.814 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:15:46.814 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:15:46.815 INFO:teuthology.orchestra.run.smithi027.stdout:Install 3 Packages 2023-11-30T21:15:46.815 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:15:46.816 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 90 M 2023-11-30T21:15:46.816 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 96 M 2023-11-30T21:15:46.816 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:15:49.566 INFO:teuthology.orchestra.run.smithi027.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 3.7 MB/s | 10 MB 00:02 2023-11-30T21:16:05.423 INFO:teuthology.orchestra.run.smithi027.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 1.9 MB/s | 36 MB 00:18 2023-11-30T21:16:21.239 INFO:teuthology.orchestra.run.smithi027.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 1.3 MB/s | 43 MB 00:34 2023-11-30T21:16:21.239 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:16:21.239 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.6 MB/s | 90 MB 00:34 2023-11-30T21:16:21.941 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:16:22.382 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:16:22.383 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:16:23.680 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:16:23.707 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:16:26.677 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:16:27.312 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-11-30T21:16:31.039 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-11-30T21:16:31.386 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-11-30T21:16:35.062 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-11-30T21:16:35.339 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-11-30T21:16:58.548 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-11-30T21:16:58.572 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-11-30T21:16:58.924 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-11-30T21:16:58.924 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-11-30T21:16:58.924 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-11-30T21:16:59.729 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-11-30T21:16:59.729 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:16:59.729 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:16:59.729 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-11-30T21:16:59.729 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-11-30T21:16:59.730 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:16:59.730 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:16:59.875 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2023-11-30T21:17:00.103 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-4.18.0-526.el8.x86_64 2023-11-30T21:17:00.103 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-11-30T21:17:00.103 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-11-30T21:17:00.103 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-30T21:17:00.104 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-30T21:17:00.104 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-11-30T21:17:00.104 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=None, version=distro) 2023-11-30T21:17:00.104 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2023-11-30T21:17:00.323 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-4.18.0-526.el8.x86_64 2023-11-30T21:17:00.323 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-11-30T21:17:00.323 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2023-11-30T21:17:00.358 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2023-11-30T21:17:00.358 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2023-11-30T21:17:00.358 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.02 2023-11-30T21:17:00.358 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 129.el8 2023-11-30T21:17:00.358 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2023-11-30T21:17:00.358 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-30T21:17:00.358 INFO:teuthology.orchestra.run.smithi027.stdout:Group : System Environment/Base 2023-11-30T21:17:00.358 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 9265586 2023-11-30T21:17:00.359 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2023-11-30T21:17:00.359 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-30T21:17:00.359 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-30T21:17:00.359 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-30T21:17:00.359 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-30T21:17:00.359 INFO:teuthology.orchestra.run.smithi027.stdout:Relocations : (not relocatable) 2023-11-30T21:17:00.359 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : CentOS BuildSystem 2023-11-30T21:17:00.359 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2023-11-30T21:17:00.359 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-30T21:17:00.359 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2023-11-30T21:17:00.359 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2023-11-30T21:17:00.360 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:17:00.360 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-30T21:17:00.360 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-30T21:17:00.360 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-30T21:17:00.360 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2023-11-30T21:17:00.360 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:17:00.360 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2023-11-30T21:17:00.361 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-30T21:17:00.361 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 4.18.0-526.el8.x86_64 2023-11-30T21:17:00.361 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-30T21:17:01.615 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2023-11-30T21:17:03.651 INFO:teuthology.orchestra.run.smithi027.stderr:done 2023-11-30T21:17:03.653 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2023-11-30T21:17:03.686 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.GdlLRUH6pv 2023-11-30T21:17:03.687 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.GdlLRUH6pv 2023-11-30T21:17:03.759 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.GdlLRUH6pv 2023-11-30T21:17:03.897 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.GdlLRUH6pv is 6KB 2023-11-30T21:17:03.945 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.GdlLRUH6pv 2023-11-30T21:17:03.963 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2023-11-30T21:17:04.038 INFO:teuthology.orchestra.run.smithi027.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-30T21:17:04.038 INFO:teuthology.orchestra.run.smithi027.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-30T21:17:04.039 INFO:teuthology.orchestra.run.smithi027.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-30T21:17:04.039 INFO:teuthology.orchestra.run.smithi027.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-11-30T21:17:04.040 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-11-30T21:17:04.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2023-11-30T21:17:34.198 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-11-30T21:17:34.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-11-30T21:17:34.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:17:52.610 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-11-30T21:18:01.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-11-30T21:18:01.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:18:04.674 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-11-30T21:18:16.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-11-30T21:18:16.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:18:19.746 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-11-30T21:18:34.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-11-30T21:18:34.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:18:37.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-11-30T21:18:55.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-11-30T21:18:55.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:19:14.274 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-11-30T21:19:35.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-11-30T21:19:35.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:19:36.515 DEBUG:teuthology.orchestra.run.smithi027:> true 2023-11-30T21:19:37.083 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-11-30T21:19:37.084 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-526.el8.x86_64"... 2023-11-30T21:19:37.084 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2023-11-30T21:19:37.105 INFO:teuthology.orchestra.run.smithi027.stdout:4.18.0-526.el8.x86_64 2023-11-30T21:19:37.107 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-11-30T21:19:37.108 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-30T21:19:37.108 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-30T21:19:38.110 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-11-30T21:19:38.110 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-30T21:19:38.194 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2023-11-30T21:19:38.210 DEBUG:teuthology.parallel:result is None 2023-11-30T21:26:25.244 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 25 kB/s | 16 MB 10:56 2023-11-30T21:26:29.276 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 398 kB/s | 24 kB 00:00 2023-11-30T21:26:33.142 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-30T21:26:33.143 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-30T21:26:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:26:33.429 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:26:33.430 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-11-30T21:26:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:26:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:26:33.430 INFO:teuthology.orchestra.run.smithi049.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-11-30T21:26:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-30T21:26:33.430 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-11-30T21:26:33.430 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-11-30T21:26:33.430 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:26:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:26:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:26:33.430 INFO:teuthology.orchestra.run.smithi049.stdout:Install 3 Packages 2023-11-30T21:26:33.431 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:26:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 90 M 2023-11-30T21:26:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 96 M 2023-11-30T21:26:33.431 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:26:43.599 INFO:teuthology.orchestra.run.smithi049.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 1.0 MB/s | 10 MB 00:10 2023-11-30T21:27:30.953 INFO:teuthology.orchestra.run.smithi049.stdout:(2/3): kernel-core-4.18.0-526.el8.x86_64.rpm 774 kB/s | 43 MB 00:57 2023-11-30T21:27:36.169 INFO:teuthology.orchestra.run.smithi049.stdout:(3/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 590 kB/s | 36 MB 01:02 2023-11-30T21:27:36.169 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:27:36.169 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.4 MB/s | 90 MB 01:02 2023-11-30T21:27:36.877 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:27:37.332 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:27:37.332 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:27:38.641 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:27:38.678 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:27:41.692 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:27:42.211 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-11-30T21:27:45.957 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-11-30T21:27:46.393 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-11-30T21:27:50.067 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-11-30T21:27:50.369 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-11-30T21:28:13.729 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-11-30T21:28:13.753 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-11-30T21:28:14.120 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-11-30T21:28:14.120 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-11-30T21:28:14.120 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-11-30T21:28:14.952 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-11-30T21:28:14.952 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:28:14.952 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:28:14.953 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-11-30T21:28:14.953 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-11-30T21:28:14.953 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:28:14.953 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:28:15.111 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2023-11-30T21:28:15.343 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-4.18.0-526.el8.x86_64 2023-11-30T21:28:15.343 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-11-30T21:28:15.343 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-11-30T21:28:15.344 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-30T21:28:15.344 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-30T21:28:15.344 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-11-30T21:28:15.344 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi049.front.sepia.ceph.com, path=None, version=distro) 2023-11-30T21:28:15.344 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2023-11-30T21:28:15.563 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-4.18.0-526.el8.x86_64 2023-11-30T21:28:15.563 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-11-30T21:28:15.563 DEBUG:teuthology.orchestra.run.smithi049:> sudo rpm -qi grub2-tools 2023-11-30T21:28:15.598 INFO:teuthology.orchestra.run.smithi049.stdout:Name : grub2-tools 2023-11-30T21:28:15.598 INFO:teuthology.orchestra.run.smithi049.stdout:Epoch : 1 2023-11-30T21:28:15.599 INFO:teuthology.orchestra.run.smithi049.stdout:Version : 2.02 2023-11-30T21:28:15.599 INFO:teuthology.orchestra.run.smithi049.stdout:Release : 129.el8 2023-11-30T21:28:15.599 INFO:teuthology.orchestra.run.smithi049.stdout:Architecture: x86_64 2023-11-30T21:28:15.599 INFO:teuthology.orchestra.run.smithi049.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-30T21:28:15.599 INFO:teuthology.orchestra.run.smithi049.stdout:Group : System Environment/Base 2023-11-30T21:28:15.599 INFO:teuthology.orchestra.run.smithi049.stdout:Size : 9265586 2023-11-30T21:28:15.599 INFO:teuthology.orchestra.run.smithi049.stdout:License : GPLv3+ 2023-11-30T21:28:15.599 INFO:teuthology.orchestra.run.smithi049.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-30T21:28:15.599 INFO:teuthology.orchestra.run.smithi049.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-30T21:28:15.599 INFO:teuthology.orchestra.run.smithi049.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-30T21:28:15.599 INFO:teuthology.orchestra.run.smithi049.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-30T21:28:15.599 INFO:teuthology.orchestra.run.smithi049.stdout:Relocations : (not relocatable) 2023-11-30T21:28:15.600 INFO:teuthology.orchestra.run.smithi049.stdout:Packager : CentOS BuildSystem 2023-11-30T21:28:15.600 INFO:teuthology.orchestra.run.smithi049.stdout:Vendor : CentOS 2023-11-30T21:28:15.600 INFO:teuthology.orchestra.run.smithi049.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-30T21:28:15.600 INFO:teuthology.orchestra.run.smithi049.stdout:Summary : Support tools for GRUB. 2023-11-30T21:28:15.600 INFO:teuthology.orchestra.run.smithi049.stdout:Description : 2023-11-30T21:28:15.600 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:28:15.600 INFO:teuthology.orchestra.run.smithi049.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-30T21:28:15.600 INFO:teuthology.orchestra.run.smithi049.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-30T21:28:15.600 INFO:teuthology.orchestra.run.smithi049.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-30T21:28:15.600 INFO:teuthology.orchestra.run.smithi049.stdout:hardware devices. 2023-11-30T21:28:15.600 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:28:15.600 INFO:teuthology.orchestra.run.smithi049.stdout:This subpackage provides tools for support of all platforms. 2023-11-30T21:28:15.601 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-30T21:28:15.601 INFO:teuthology.task.kernel:Updating grub on smithi049 to boot 4.18.0-526.el8.x86_64 2023-11-30T21:28:15.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-30T21:28:17.101 INFO:teuthology.orchestra.run.smithi049.stderr:Generating grub configuration file ... 2023-11-30T21:28:19.284 INFO:teuthology.orchestra.run.smithi049.stderr:done 2023-11-30T21:28:19.286 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2023-11-30T21:28:19.316 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.5juGlMZJB8 2023-11-30T21:28:19.316 DEBUG:teuthology.orchestra.run.smithi049:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5juGlMZJB8 2023-11-30T21:28:19.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.5juGlMZJB8 2023-11-30T21:28:19.567 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.5juGlMZJB8 is 6KB 2023-11-30T21:28:19.599 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.5juGlMZJB8 2023-11-30T21:28:19.620 DEBUG:teuthology.orchestra.run.smithi049:> sudo /bin/ls /boot/loader/entries || true 2023-11-30T21:28:19.700 INFO:teuthology.orchestra.run.smithi049.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-30T21:28:19.701 INFO:teuthology.orchestra.run.smithi049.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-30T21:28:19.701 INFO:teuthology.orchestra.run.smithi049.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-30T21:28:19.701 INFO:teuthology.orchestra.run.smithi049.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-11-30T21:28:19.702 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-11-30T21:28:19.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo shutdown -r now 2023-11-30T21:28:49.893 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-11-30T21:28:49.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-30T21:28:49.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:29:24.578 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-11-30T21:29:33.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-30T21:29:33.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:29:36.642 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-11-30T21:29:48.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-30T21:29:48.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:29:51.714 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-11-30T21:30:06.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-30T21:30:06.716 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:30:09.794 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-11-30T21:30:27.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-30T21:30:27.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:30:30.850 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2023-11-30T21:30:51.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-30T21:30:51.852 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:30:52.455 DEBUG:teuthology.orchestra.run.smithi049:> true 2023-11-30T21:30:52.780 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-30T21:30:52.780 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-526.el8.x86_64"... 2023-11-30T21:30:52.780 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2023-11-30T21:30:52.800 INFO:teuthology.orchestra.run.smithi049.stdout:4.18.0-526.el8.x86_64 2023-11-30T21:30:52.801 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-11-30T21:30:52.801 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-30T21:30:52.801 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-30T21:30:53.802 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-11-30T21:30:53.802 DEBUG:teuthology.orchestra.run.smithi049:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-30T21:30:53.871 INFO:teuthology.orchestra.run.smithi049.stdout:ttyS1 2023-11-30T21:30:53.883 DEBUG:teuthology.parallel:result is None 2023-11-30T21:30:53.884 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-30T21:30:53.890 INFO:teuthology.task.internal:Creating test directory... 2023-11-30T21:30:53.890 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-30T21:30:53.893 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-30T21:30:53.913 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-30T21:30:53.920 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-30T21:30:53.934 INFO:teuthology.task.internal:Creating archive directory... 2023-11-30T21:30:53.934 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-30T21:30:53.956 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-30T21:30:54.007 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-30T21:30:54.014 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-30T21:30:54.014 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-30T21:30:54.016 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-30T21:30:54.051 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-30T21:30:54.059 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-30T21:30:54.062 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-30T21:30:54.070 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-30T21:30:54.072 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-30T21:30:54.079 INFO:teuthology.task.internal:Configuring sudo... 2023-11-30T21:30:54.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-30T21:30:54.106 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-30T21:30:54.134 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-30T21:30:54.143 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-30T21:30:54.143 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-30T21:30:54.177 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-30T21:30:54.196 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-30T21:30:54.285 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-30T21:30:54.350 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-30T21:30:54.394 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-30T21:30:54.439 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-11-30T21:30:54.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-30T21:30:54.508 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-30T21:30:54.555 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-30T21:30:54.600 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-30T21:30:54.684 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-30T21:30:54.729 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-30T21:30:54.730 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-30T21:30:54.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2023-11-30T21:30:54.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo service rsyslog restart 2023-11-30T21:30:54.873 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-30T21:30:54.937 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-30T21:30:55.274 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-30T21:30:55.283 INFO:teuthology.task.internal:Starting timer... 2023-11-30T21:30:55.283 INFO:teuthology.run_tasks:Running task pcp... 2023-11-30T21:30:55.292 INFO:teuthology.run_tasks:Running task selinux... 2023-11-30T21:30:55.300 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-30T21:30:55.301 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2023-11-30T21:30:55.418 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2023-11-30T21:30:55.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo service auditd rotate 2023-11-30T21:30:55.503 INFO:teuthology.orchestra.run.smithi049.stdout:Rotating logs: 2023-11-30T21:30:55.505 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-30T21:30:55.505 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2023-11-30T21:30:55.530 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2023-11-30T21:30:55.530 DEBUG:teuthology.orchestra.run.smithi049:> /usr/sbin/getenforce 2023-11-30T21:30:55.556 INFO:teuthology.orchestra.run.smithi049.stdout:Permissive 2023-11-30T21:30:55.556 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi049.front.sepia.ceph.com': 'permissive'} 2023-11-30T21:30:55.557 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-30T21:30:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-30T21:30:55.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-30T21:30:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-30T21:30:55.630 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-30T21:30:55.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2023-11-30T21:30:55.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo /usr/sbin/setenforce permissive 2023-11-30T21:30:55.699 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-30T21:30:55.709 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-30T21:30:55.726 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-30T21:30:55.727 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi049.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-30T21:38:38.784 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi049.front.sepia.ceph.com')] 2023-11-30T21:38:38.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-11-30T21:38:38.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:38:38.856 DEBUG:teuthology.orchestra.run.smithi027:> true 2023-11-30T21:38:38.925 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-11-30T21:38:38.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-30T21:38:38.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-30T21:38:38.995 DEBUG:teuthology.orchestra.run.smithi049:> true 2023-11-30T21:38:39.068 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2023-11-30T21:38:39.069 INFO:teuthology.run_tasks:Running task clock... 2023-11-30T21:38:39.080 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-30T21:38:39.080 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-30T21:38:39.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-30T21:38:39.083 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-30T21:38:39.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-30T21:38:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-30T21:38:39.130 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-30T21:38:39.142 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-30T21:38:39.156 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-30T21:38:39.200 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2023-11-30T21:38:39.212 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2023-11-30T21:38:39.217 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: ntpd: command not found 2023-11-30T21:38:39.226 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-30T21:38:39.228 INFO:teuthology.orchestra.run.smithi049.stdout:506 Cannot talk to daemon 2023-11-30T21:38:39.240 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-30T21:38:39.243 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-30T21:38:39.257 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-30T21:38:39.308 INFO:teuthology.orchestra.run.smithi027.stderr:bash: ntpq: command not found 2023-11-30T21:38:39.312 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-30T21:38:39.312 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2023-11-30T21:38:39.312 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-30T21:38:39.312 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-30T21:38:39.312 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-30T21:38:39.312 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-30T21:38:39.335 INFO:teuthology.orchestra.run.smithi049.stderr:bash: ntpq: command not found 2023-11-30T21:38:39.339 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-30T21:38:39.340 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2023-11-30T21:38:39.340 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-30T21:38:39.340 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-30T21:38:39.340 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-30T21:38:39.340 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-30T21:38:39.341 INFO:teuthology.run_tasks:Running task pexec... 2023-11-30T21:38:39.351 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-30T21:38:39.351 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi027.front.sepia.ceph.com 2023-11-30T21:38:39.351 DEBUG:teuthology.orchestra.run.smithi027:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-30T21:38:39.352 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi049.front.sepia.ceph.com 2023-11-30T21:38:39.352 DEBUG:teuthology.orchestra.run.smithi049:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-30T21:38:40.269 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:42 ago on Thu 30 Nov 2023 09:36:58 PM UTC. 2023-11-30T21:38:40.287 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:43 ago on Thu 30 Nov 2023 09:36:57 PM UTC. 2023-11-30T21:38:42.564 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:38:42.565 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:38:42.565 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2023-11-30T21:38:42.565 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:38:42.566 INFO:teuthology.orchestra.run.smithi027.stdout:Resetting modules: 2023-11-30T21:38:42.566 INFO:teuthology.orchestra.run.smithi027.stdout: container-tools 2023-11-30T21:38:42.566 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:38:42.566 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:38:42.566 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:38:42.566 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:38:42.589 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:38:42.590 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:38:42.590 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-11-30T21:38:42.590 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:38:42.590 INFO:teuthology.orchestra.run.smithi049.stdout:Resetting modules: 2023-11-30T21:38:42.590 INFO:teuthology.orchestra.run.smithi049.stdout: container-tools 2023-11-30T21:38:42.590 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:38:42.591 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:38:42.591 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:38:42.591 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:38:42.836 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:38:42.854 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:38:43.815 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:45 ago on Thu 30 Nov 2023 09:36:58 PM UTC. 2023-11-30T21:38:43.825 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:46 ago on Thu 30 Nov 2023 09:36:57 PM UTC. 2023-11-30T21:38:46.073 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:38:46.098 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:38:46.271 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:38:46.271 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-30T21:38:46.272 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:38:46.272 INFO:teuthology.orchestra.run.smithi049.stdout:Installing group/module packages: 2023-11-30T21:38:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-30T21:38:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-30T21:38:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-30T21:38:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-30T21:38:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-30T21:38:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-30T21:38:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-30T21:38:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-30T21:38:46.272 INFO:teuthology.orchestra.run.smithi049.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-30T21:38:46.272 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-30T21:38:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-30T21:38:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-30T21:38:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-30T21:38:46.273 INFO:teuthology.orchestra.run.smithi049.stdout:Installing weak dependencies: 2023-11-30T21:38:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-30T21:38:46.273 INFO:teuthology.orchestra.run.smithi049.stdout:Installing module profiles: 2023-11-30T21:38:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: container-tools/common 2023-11-30T21:38:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:38:46.273 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling module streams: 2023-11-30T21:38:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: container-tools rhel8 2023-11-30T21:38:46.273 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:38:46.273 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:38:46.274 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:38:46.274 INFO:teuthology.orchestra.run.smithi049.stdout:Install 13 Packages 2023-11-30T21:38:46.274 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:38:46.274 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 27 M 2023-11-30T21:38:46.274 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 89 M 2023-11-30T21:38:46.274 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:38:46.302 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:38:46.302 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-11-30T21:38:46.302 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:38:46.302 INFO:teuthology.orchestra.run.smithi027.stdout:Installing group/module packages: 2023-11-30T21:38:46.302 INFO:teuthology.orchestra.run.smithi027.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-30T21:38:46.302 INFO:teuthology.orchestra.run.smithi027.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-30T21:38:46.303 INFO:teuthology.orchestra.run.smithi027.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-30T21:38:46.303 INFO:teuthology.orchestra.run.smithi027.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-30T21:38:46.303 INFO:teuthology.orchestra.run.smithi027.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-30T21:38:46.303 INFO:teuthology.orchestra.run.smithi027.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-30T21:38:46.303 INFO:teuthology.orchestra.run.smithi027.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-30T21:38:46.303 INFO:teuthology.orchestra.run.smithi027.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-30T21:38:46.303 INFO:teuthology.orchestra.run.smithi027.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-30T21:38:46.303 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-11-30T21:38:46.303 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-30T21:38:46.303 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-30T21:38:46.303 INFO:teuthology.orchestra.run.smithi027.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-30T21:38:46.303 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2023-11-30T21:38:46.304 INFO:teuthology.orchestra.run.smithi027.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-30T21:38:46.304 INFO:teuthology.orchestra.run.smithi027.stdout:Installing module profiles: 2023-11-30T21:38:46.304 INFO:teuthology.orchestra.run.smithi027.stdout: container-tools/common 2023-11-30T21:38:46.304 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:38:46.304 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling module streams: 2023-11-30T21:38:46.304 INFO:teuthology.orchestra.run.smithi027.stdout: container-tools rhel8 2023-11-30T21:38:46.304 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:38:46.304 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:38:46.304 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:38:46.304 INFO:teuthology.orchestra.run.smithi027.stdout:Install 13 Packages 2023-11-30T21:38:46.304 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:38:46.305 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 27 M 2023-11-30T21:38:46.305 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 89 M 2023-11-30T21:38:46.305 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:38:47.052 INFO:teuthology.orchestra.run.smithi049.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.2 MB/s | 750 kB 00:00 2023-11-30T21:38:47.075 INFO:teuthology.orchestra.run.smithi027.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.3 MB/s | 750 kB 00:00 2023-11-30T21:38:47.111 INFO:teuthology.orchestra.run.smithi049.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 1.5 MB/s | 1.0 MB 00:00 2023-11-30T21:38:47.153 INFO:teuthology.orchestra.run.smithi049.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 377 kB/s | 38 kB 00:00 2023-11-30T21:38:47.167 INFO:teuthology.orchestra.run.smithi027.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 1.5 MB/s | 1.0 MB 00:00 2023-11-30T21:38:47.192 INFO:teuthology.orchestra.run.smithi027.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 324 kB/s | 38 kB 00:00 2023-11-30T21:38:47.220 INFO:teuthology.orchestra.run.smithi049.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.2 MB/s | 241 kB 00:00 2023-11-30T21:38:47.276 INFO:teuthology.orchestra.run.smithi027.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.2 MB/s | 241 kB 00:00 2023-11-30T21:38:47.337 INFO:teuthology.orchestra.run.smithi049.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.5 MB/s | 178 kB 00:00 2023-11-30T21:38:47.409 INFO:teuthology.orchestra.run.smithi027.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.3 MB/s | 178 kB 00:00 2023-11-30T21:38:47.446 INFO:teuthology.orchestra.run.smithi049.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 229 kB/s | 25 kB 00:00 2023-11-30T21:38:47.527 INFO:teuthology.orchestra.run.smithi027.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 212 kB/s | 25 kB 00:00 2023-11-30T21:38:47.563 INFO:teuthology.orchestra.run.smithi049.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 805 kB/s | 94 kB 00:00 2023-11-30T21:38:47.635 INFO:teuthology.orchestra.run.smithi027.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 866 kB/s | 94 kB 00:00 2023-11-30T21:38:47.680 INFO:teuthology.orchestra.run.smithi049.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 7.4 MB/s | 3.9 MB 00:00 2023-11-30T21:38:47.735 INFO:teuthology.orchestra.run.smithi027.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 7.2 MB/s | 3.9 MB 00:00 2023-11-30T21:38:47.930 INFO:teuthology.orchestra.run.smithi049.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 6.5 MB/s | 9.6 MB 00:01 2023-11-30T21:38:47.978 INFO:teuthology.orchestra.run.smithi027.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 6.5 MB/s | 9.6 MB 00:01 2023-11-30T21:38:48.049 INFO:teuthology.orchestra.run.smithi049.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 6.9 MB/s | 2.5 MB 00:00 2023-11-30T21:38:48.072 INFO:teuthology.orchestra.run.smithi049.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 371 kB/s | 53 kB 00:00 2023-11-30T21:38:48.086 INFO:teuthology.orchestra.run.smithi027.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 487 kB/s | 53 kB 00:00 2023-11-30T21:38:48.180 INFO:teuthology.orchestra.run.smithi027.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 5.7 MB/s | 2.5 MB 00:00 2023-11-30T21:38:48.181 INFO:teuthology.orchestra.run.smithi049.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 306 kB/s | 41 kB 00:00 2023-11-30T21:38:48.212 INFO:teuthology.orchestra.run.smithi027.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 327 kB/s | 41 kB 00:00 2023-11-30T21:38:48.448 INFO:teuthology.orchestra.run.smithi049.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 9.2 MB/s | 8.1 MB 00:00 2023-11-30T21:38:48.448 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:38:48.448 INFO:teuthology.orchestra.run.smithi049.stdout:Total 12 MB/s | 27 MB 00:02 2023-11-30T21:38:48.634 INFO:teuthology.orchestra.run.smithi027.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 8.2 MB/s | 8.1 MB 00:00 2023-11-30T21:38:48.634 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:38:48.635 INFO:teuthology.orchestra.run.smithi027.stdout:Total 11 MB/s | 27 MB 00:02 2023-11-30T21:38:48.724 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:38:48.747 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:38:48.747 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:38:48.916 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:38:48.939 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:38:48.939 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:38:49.011 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:38:49.014 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:38:49.216 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:38:49.219 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:38:49.745 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:38:49.978 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-30T21:38:49.994 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:38:50.130 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-30T21:38:50.196 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-30T21:38:50.334 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-30T21:38:50.399 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-30T21:38:50.577 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-30T21:38:50.691 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-30T21:38:50.914 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-30T21:38:51.255 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-30T21:38:51.455 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-30T21:38:52.281 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-30T21:38:52.494 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-30T21:38:52.528 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-30T21:38:52.695 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-30T21:38:52.794 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-30T21:38:52.984 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-30T21:38:53.021 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-30T21:38:53.287 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-30T21:38:53.397 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-30T21:38:53.716 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-30T21:38:54.328 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-30T21:38:54.582 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-30T21:38:54.662 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-30T21:38:54.762 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-30T21:38:54.946 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-30T21:38:55.164 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-30T21:38:55.206 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-30T21:38:55.207 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-30T21:38:55.207 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-30T21:38:55.207 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-30T21:38:55.207 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-30T21:38:55.207 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-30T21:38:55.207 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-30T21:38:55.207 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-30T21:38:55.207 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-30T21:38:55.207 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-30T21:38:55.207 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-30T21:38:55.207 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-30T21:38:55.208 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-30T21:38:55.624 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-30T21:38:55.624 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-30T21:38:55.624 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-30T21:38:55.624 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-30T21:38:55.624 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-30T21:38:55.625 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-30T21:38:55.625 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-30T21:38:55.625 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-30T21:38:55.625 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-30T21:38:55.625 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-30T21:38:55.625 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-30T21:38:55.625 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-30T21:38:55.625 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-30T21:38:56.101 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-30T21:38:56.101 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:38:56.101 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:38:56.101 INFO:teuthology.orchestra.run.smithi049.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-30T21:38:56.101 INFO:teuthology.orchestra.run.smithi049.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-30T21:38:56.101 INFO:teuthology.orchestra.run.smithi049.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-30T21:38:56.101 INFO:teuthology.orchestra.run.smithi049.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-30T21:38:56.102 INFO:teuthology.orchestra.run.smithi049.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-30T21:38:56.102 INFO:teuthology.orchestra.run.smithi049.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-30T21:38:56.102 INFO:teuthology.orchestra.run.smithi049.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-30T21:38:56.102 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-30T21:38:56.102 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-30T21:38:56.102 INFO:teuthology.orchestra.run.smithi049.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-30T21:38:56.102 INFO:teuthology.orchestra.run.smithi049.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-30T21:38:56.102 INFO:teuthology.orchestra.run.smithi049.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-30T21:38:56.102 INFO:teuthology.orchestra.run.smithi049.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-30T21:38:56.102 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:38:56.102 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:38:56.358 DEBUG:teuthology.parallel:result is None 2023-11-30T21:38:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-30T21:38:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:38:56.494 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:38:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-30T21:38:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-30T21:38:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-30T21:38:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-30T21:38:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-30T21:38:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-30T21:38:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-30T21:38:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-30T21:38:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-30T21:38:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-30T21:38:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-30T21:38:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-30T21:38:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-30T21:38:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:38:56.495 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:38:56.768 DEBUG:teuthology.parallel:result is None 2023-11-30T21:38:56.807 INFO:teuthology.run_tasks:Running task pexec... 2023-11-30T21:38:56.894 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-30T21:38:56.894 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi027.front.sepia.ceph.com 2023-11-30T21:38:56.895 DEBUG:teuthology.orchestra.run.smithi027:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-30T21:38:56.895 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi049.front.sepia.ceph.com 2023-11-30T21:38:56.895 DEBUG:teuthology.orchestra.run.smithi049:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-30T21:38:57.042 DEBUG:teuthology.parallel:result is None 2023-11-30T21:38:57.143 DEBUG:teuthology.parallel:result is None 2023-11-30T21:38:57.143 INFO:teuthology.run_tasks:Running task install... 2023-11-30T21:38:57.153 DEBUG:teuthology.task.install:project ceph 2023-11-30T21:38:57.153 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9'}} 2023-11-30T21:38:57.153 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9'} 2023-11-30T21:38:57.153 INFO:teuthology.task.install:Using flavor: default 2023-11-30T21:38:57.159 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-30T21:38:57.159 INFO:teuthology.task.install:extra packages: [] 2023-11-30T21:38:57.160 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-30T21:38:57.160 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-30T21:38:57.160 INFO:teuthology.packaging:ref: None 2023-11-30T21:38:57.160 INFO:teuthology.packaging:tag: v16.2.4 2023-11-30T21:38:57.160 INFO:teuthology.packaging:branch: None 2023-11-30T21:38:57.160 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-11-30T21:38:57.425 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-30T21:38:57.427 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-30T21:38:57.429 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-30T21:38:57.429 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-30T21:38:57.429 INFO:teuthology.packaging:ref: None 2023-11-30T21:38:57.430 INFO:teuthology.packaging:tag: v16.2.4 2023-11-30T21:38:57.430 INFO:teuthology.packaging:branch: None 2023-11-30T21:38:57.430 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-11-30T21:38:57.430 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-30T21:38:57.814 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-30T21:38:57.815 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-30T21:38:57.822 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-30T21:38:57.822 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-30T21:38:57.855 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-30T21:38:57.855 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-11-30T21:38:57.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-30T21:38:57.858 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-30T21:38:57.858 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-30T21:38:57.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-30T21:38:57.892 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-30T21:38:57.893 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-30T21:38:57.893 INFO:teuthology.packaging:ref: None 2023-11-30T21:38:57.893 INFO:teuthology.packaging:tag: v16.2.4 2023-11-30T21:38:57.893 INFO:teuthology.packaging:branch: None 2023-11-30T21:38:57.893 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-11-30T21:38:57.893 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-30T21:38:57.894 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-30T21:38:57.894 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-30T21:38:57.895 INFO:teuthology.packaging:ref: None 2023-11-30T21:38:57.895 INFO:teuthology.packaging:tag: v16.2.4 2023-11-30T21:38:57.895 INFO:teuthology.packaging:branch: None 2023-11-30T21:38:57.895 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-11-30T21:38:57.895 DEBUG:teuthology.orchestra.run.smithi049:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-30T21:38:57.967 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-30T21:38:57.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-30T21:38:58.049 DEBUG:teuthology.orchestra.run.smithi049:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-30T21:38:58.050 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-30T21:38:58.118 INFO:teuthology.orchestra.run.smithi049.stdout:check_obsoletes = 1 2023-11-30T21:38:58.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum clean all 2023-11-30T21:38:58.121 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2023-11-30T21:38:58.123 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2023-11-30T21:38:58.404 INFO:teuthology.orchestra.run.smithi027.stdout:76 files removed 2023-11-30T21:38:58.411 INFO:teuthology.orchestra.run.smithi049.stdout:76 files removed 2023-11-30T21:38:58.434 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw 2023-11-30T21:38:58.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-radosgw 2023-11-30T21:38:58.958 INFO:teuthology.orchestra.run.smithi049.stdout:ceph packages for x86_64 890 kB/s | 74 kB 00:00 2023-11-30T21:38:58.964 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 738 kB/s | 74 kB 00:00 2023-11-30T21:38:59.081 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 197 kB/s | 15 kB 00:00 2023-11-30T21:38:59.130 INFO:teuthology.orchestra.run.smithi049.stdout:ceph noarch packages 116 kB/s | 15 kB 00:00 2023-11-30T21:38:59.198 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 20 kB/s | 1.7 kB 00:00 2023-11-30T21:38:59.251 INFO:teuthology.orchestra.run.smithi049.stdout:ceph source packages 19 kB/s | 1.7 kB 00:00 2023-11-30T21:39:02.526 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - AppStream 11 MB/s | 34 MB 00:03 2023-11-30T21:39:02.741 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - AppStream 9.7 MB/s | 34 MB 00:03 2023-11-30T21:39:13.316 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 55 MB 00:03 2023-11-30T21:39:14.251 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 55 MB 00:04 2023-11-30T21:39:22.032 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - Extras 50 kB/s | 18 kB 00:00 2023-11-30T21:39:22.616 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2023-11-30T21:39:22.910 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - Extras 51 kB/s | 18 kB 00:00 2023-11-30T21:39:23.469 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-11-30T21:39:24.070 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 8 - PowerTools 15 MB/s | 6.6 MB 00:00 2023-11-30T21:39:24.168 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 8 - PowerTools 4.8 MB/s | 6.6 MB 00:01 2023-11-30T21:39:25.604 INFO:teuthology.orchestra.run.smithi049.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-11-30T21:39:25.678 INFO:teuthology.orchestra.run.smithi027.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-11-30T21:39:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 16 MB 00:02 2023-11-30T21:39:29.358 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 16 MB 00:03 2023-11-30T21:39:31.996 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 475 kB/s | 24 kB 00:00 2023-11-30T21:39:33.447 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 259 kB/s | 24 kB 00:00 2023-11-30T21:39:36.888 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:39:36.891 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:39:36.891 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-11-30T21:39:36.892 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:39:36.892 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:39:36.892 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-30T21:39:36.892 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2023-11-30T21:39:36.892 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-30T21:39:36.892 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-30T21:39:36.892 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-11-30T21:39:36.892 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-30T21:39:36.892 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-30T21:39:36.892 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-30T21:39:36.892 INFO:teuthology.orchestra.run.smithi027.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-30T21:39:36.892 INFO:teuthology.orchestra.run.smithi027.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-30T21:39:36.893 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-30T21:39:36.893 INFO:teuthology.orchestra.run.smithi027.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-30T21:39:36.893 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-30T21:39:36.893 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-30T21:39:36.893 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-30T21:39:36.893 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-30T21:39:36.893 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-30T21:39:36.893 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-30T21:39:36.893 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-30T21:39:36.893 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-30T21:39:36.893 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-30T21:39:36.894 INFO:teuthology.orchestra.run.smithi027.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-30T21:39:36.894 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-30T21:39:36.894 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-30T21:39:36.894 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-30T21:39:36.894 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-30T21:39:36.894 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-30T21:39:36.894 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-30T21:39:36.894 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-30T21:39:36.894 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:39:36.894 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:39:36.894 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:39:36.894 INFO:teuthology.orchestra.run.smithi027.stdout:Install 25 Packages 2023-11-30T21:39:36.895 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 2 Packages 2023-11-30T21:39:36.895 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:39:36.895 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 59 M 2023-11-30T21:39:36.895 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:39:37.331 INFO:teuthology.orchestra.run.smithi027.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 40 MB/s | 6.0 MB 00:00 2023-11-30T21:39:37.348 INFO:teuthology.orchestra.run.smithi027.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2023-11-30T21:39:37.381 INFO:teuthology.orchestra.run.smithi027.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2023-11-30T21:39:37.415 INFO:teuthology.orchestra.run.smithi027.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-11-30T21:39:37.565 INFO:teuthology.orchestra.run.smithi027.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 27 MB/s | 4.1 MB 00:00 2023-11-30T21:39:37.590 INFO:teuthology.orchestra.run.smithi027.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-11-30T21:39:37.616 INFO:teuthology.orchestra.run.smithi027.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-30T21:39:37.641 INFO:teuthology.orchestra.run.smithi027.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2023-11-30T21:39:37.675 INFO:teuthology.orchestra.run.smithi027.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-30T21:39:37.736 INFO:teuthology.orchestra.run.smithi027.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 5.9 MB/s | 368 kB 00:00 2023-11-30T21:39:37.776 INFO:teuthology.orchestra.run.smithi027.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 2.9 MB/s | 114 kB 00:00 2023-11-30T21:39:37.976 INFO:teuthology.orchestra.run.smithi027.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 16 MB/s | 13 MB 00:00 2023-11-30T21:39:38.301 INFO:teuthology.orchestra.run.smithi027.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 21 MB/s | 24 MB 00:01 2023-11-30T21:39:38.315 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:39:38.318 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:39:38.318 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-30T21:39:38.319 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:39:38.319 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:39:38.319 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-30T21:39:38.319 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrading: 2023-11-30T21:39:38.319 INFO:teuthology.orchestra.run.smithi049.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-30T21:39:38.319 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-30T21:39:38.319 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-30T21:39:38.319 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-30T21:39:38.319 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-30T21:39:38.319 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-30T21:39:38.319 INFO:teuthology.orchestra.run.smithi049.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-30T21:39:38.320 INFO:teuthology.orchestra.run.smithi049.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-30T21:39:38.320 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-30T21:39:38.320 INFO:teuthology.orchestra.run.smithi049.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-30T21:39:38.320 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-30T21:39:38.320 INFO:teuthology.orchestra.run.smithi049.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-30T21:39:38.320 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-30T21:39:38.320 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-30T21:39:38.320 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-30T21:39:38.320 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-30T21:39:38.320 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-30T21:39:38.320 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-30T21:39:38.321 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-30T21:39:38.321 INFO:teuthology.orchestra.run.smithi049.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-30T21:39:38.321 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-30T21:39:38.321 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-30T21:39:38.321 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-30T21:39:38.321 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-30T21:39:38.321 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-30T21:39:38.321 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-30T21:39:38.321 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-30T21:39:38.321 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:39:38.321 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:39:38.322 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:39:38.322 INFO:teuthology.orchestra.run.smithi049.stdout:Install 25 Packages 2023-11-30T21:39:38.322 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrade 2 Packages 2023-11-30T21:39:38.322 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:39:38.322 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 59 M 2023-11-30T21:39:38.322 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:39:38.402 INFO:teuthology.orchestra.run.smithi027.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 232 kB/s | 145 kB 00:00 2023-11-30T21:39:38.519 INFO:teuthology.orchestra.run.smithi027.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 376 kB/s | 44 kB 00:00 2023-11-30T21:39:38.836 INFO:teuthology.orchestra.run.smithi027.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 133 kB/s | 42 kB 00:00 2023-11-30T21:39:38.961 INFO:teuthology.orchestra.run.smithi027.stdout:(17/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 376 kB/s | 47 kB 00:00 2023-11-30T21:39:38.986 INFO:teuthology.orchestra.run.smithi027.stdout:(18/27): librdkafka-0.11.4-3.el8.x86_64.rpm 350 kB/s | 354 kB 00:01 2023-11-30T21:39:39.166 INFO:teuthology.orchestra.run.smithi049.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 31 MB/s | 6.0 MB 00:00 2023-11-30T21:39:39.179 INFO:teuthology.orchestra.run.smithi027.stdout:(19/27): ndctl-libs-71.1-7.el8.x86_64.rpm 368 kB/s | 80 kB 00:00 2023-11-30T21:39:39.191 INFO:teuthology.orchestra.run.smithi049.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2023-11-30T21:39:39.212 INFO:teuthology.orchestra.run.smithi027.stdout:(20/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 285 kB/s | 259 kB 00:00 2023-11-30T21:39:39.266 INFO:teuthology.orchestra.run.smithi049.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 10 MB/s | 801 kB 00:00 2023-11-30T21:39:39.300 INFO:teuthology.orchestra.run.smithi049.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-11-30T21:39:39.409 INFO:teuthology.orchestra.run.smithi049.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 38 MB/s | 4.1 MB 00:00 2023-11-30T21:39:39.434 INFO:teuthology.orchestra.run.smithi049.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-11-30T21:39:39.451 INFO:teuthology.orchestra.run.smithi049.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 4.7 MB/s | 80 kB 00:00 2023-11-30T21:39:39.484 INFO:teuthology.orchestra.run.smithi049.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.3 MB/s | 213 kB 00:00 2023-11-30T21:39:39.518 INFO:teuthology.orchestra.run.smithi049.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-30T21:39:39.552 INFO:teuthology.orchestra.run.smithi049.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-11-30T21:39:39.577 INFO:teuthology.orchestra.run.smithi049.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-11-30T21:39:39.711 INFO:teuthology.orchestra.run.smithi049.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 17 MB/s | 13 MB 00:00 2023-11-30T21:39:39.770 INFO:teuthology.orchestra.run.smithi027.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 148 kB/s | 116 kB 00:00 2023-11-30T21:39:39.871 INFO:teuthology.orchestra.run.smithi027.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 275 kB/s | 181 kB 00:00 2023-11-30T21:39:39.897 INFO:teuthology.orchestra.run.smithi027.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 427 kB/s | 306 kB 00:00 2023-11-30T21:39:39.922 INFO:teuthology.orchestra.run.smithi027.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 391 kB/s | 59 kB 00:00 2023-11-30T21:39:40.002 INFO:teuthology.orchestra.run.smithi049.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 23 MB/s | 24 MB 00:01 2023-11-30T21:39:40.013 INFO:teuthology.orchestra.run.smithi027.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2023-11-30T21:39:40.038 INFO:teuthology.orchestra.run.smithi027.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 448 kB/s | 75 kB 00:00 2023-11-30T21:39:40.103 INFO:teuthology.orchestra.run.smithi049.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 276 kB/s | 145 kB 00:00 2023-11-30T21:39:40.211 INFO:teuthology.orchestra.run.smithi049.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 406 kB/s | 44 kB 00:00 2023-11-30T21:39:40.355 INFO:teuthology.orchestra.run.smithi027.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 9.3 MB/s | 4.0 MB 00:00 2023-11-30T21:39:40.360 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:39:40.360 INFO:teuthology.orchestra.run.smithi027.stdout:Total 17 MB/s | 59 MB 00:03 2023-11-30T21:39:40.430 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:39:40.453 INFO:teuthology.orchestra.run.smithi049.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 175 kB/s | 42 kB 00:00 2023-11-30T21:39:40.483 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:39:40.483 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:39:40.562 INFO:teuthology.orchestra.run.smithi049.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 464 kB/s | 259 kB 00:00 2023-11-30T21:39:40.587 INFO:teuthology.orchestra.run.smithi049.stdout:(18/27): librdkafka-0.11.4-3.el8.x86_64.rpm 404 kB/s | 354 kB 00:00 2023-11-30T21:39:40.612 INFO:teuthology.orchestra.run.smithi049.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 296 kB/s | 47 kB 00:00 2023-11-30T21:39:40.679 INFO:teuthology.orchestra.run.smithi049.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 685 kB/s | 80 kB 00:00 2023-11-30T21:39:40.763 INFO:teuthology.orchestra.run.smithi049.stdout:(21/27): gperftools-libs-2.7-9.el8.x86_64.rpm 2.0 MB/s | 306 kB 00:00 2023-11-30T21:39:40.788 INFO:teuthology.orchestra.run.smithi049.stdout:(22/27): fmt-6.2.1-1.el8.x86_64.rpm 663 kB/s | 116 kB 00:00 2023-11-30T21:39:40.813 INFO:teuthology.orchestra.run.smithi049.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 1.1 MB/s | 59 kB 00:00 2023-11-30T21:39:40.839 INFO:teuthology.orchestra.run.smithi049.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 1.1 MB/s | 181 kB 00:00 2023-11-30T21:39:40.858 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:39:40.861 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:39:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 994 kB/s | 75 kB 00:00 2023-11-30T21:39:40.998 INFO:teuthology.orchestra.run.smithi049.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 20 MB/s | 3.7 MB 00:00 2023-11-30T21:39:41.056 INFO:teuthology.orchestra.run.smithi049.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 19 MB/s | 4.0 MB 00:00 2023-11-30T21:39:41.058 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:39:41.058 INFO:teuthology.orchestra.run.smithi049.stdout:Total 21 MB/s | 59 MB 00:02 2023-11-30T21:39:41.126 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:39:41.179 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:39:41.179 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:39:41.558 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:39:41.562 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:39:42.142 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:39:42.420 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-30T21:39:42.610 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-30T21:39:42.714 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:39:42.792 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-30T21:39:42.944 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-30T21:39:43.141 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-30T21:39:43.261 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-30T21:39:43.355 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-30T21:39:43.458 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-30T21:39:43.601 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-30T21:39:43.792 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-30T21:39:43.859 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-30T21:39:44.004 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-30T21:39:44.058 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-30T21:39:44.093 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-30T21:39:44.110 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-30T21:39:44.326 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-30T21:39:44.333 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-30T21:39:44.517 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-30T21:39:44.529 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-30T21:39:44.577 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-30T21:39:44.755 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-30T21:39:44.904 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-30T21:39:44.949 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-30T21:39:45.241 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-30T21:39:45.324 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-30T21:39:45.358 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-30T21:39:45.479 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-30T21:39:45.547 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-30T21:39:45.556 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-30T21:39:45.776 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-30T21:39:45.783 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-30T21:39:45.892 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-30T21:39:45.967 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-30T21:39:46.034 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-30T21:39:46.066 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-30T21:39:46.195 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-30T21:39:46.285 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-30T21:39:46.380 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-30T21:39:46.466 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-30T21:39:46.554 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-30T21:39:46.572 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-30T21:39:46.636 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-30T21:39:46.716 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-30T21:39:46.812 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-30T21:39:47.225 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-30T21:39:47.254 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-30T21:39:47.408 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-30T21:39:47.445 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-30T21:39:47.547 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-30T21:39:47.550 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-30T21:39:47.700 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-30T21:39:47.759 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-30T21:39:47.900 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-30T21:39:47.942 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-30T21:39:48.057 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-30T21:39:48.130 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-30T21:39:48.196 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-30T21:39:48.261 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-30T21:39:48.344 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-30T21:39:48.448 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-30T21:39:48.575 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-30T21:39:48.654 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-30T21:39:48.777 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-30T21:39:48.822 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-30T21:39:49.004 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-30T21:39:49.158 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-30T21:39:51.739 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-30T21:39:51.937 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-30T21:39:51.957 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-30T21:39:52.191 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-30T21:39:52.695 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-30T21:39:52.934 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-30T21:39:52.957 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-30T21:39:53.054 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-30T21:39:53.219 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-30T21:39:53.253 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-30T21:39:53.393 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-30T21:39:53.573 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-30T21:40:12.391 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-30T21:40:12.392 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /sys 2023-11-30T21:40:12.392 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /proc 2023-11-30T21:40:12.392 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /dev 2023-11-30T21:40:12.392 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /run 2023-11-30T21:40:12.392 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /mnt 2023-11-30T21:40:12.393 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /var/tmp 2023-11-30T21:40:12.393 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /home 2023-11-30T21:40:12.393 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /tmp 2023-11-30T21:40:12.393 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /dev 2023-11-30T21:40:12.393 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:12.827 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-30T21:40:12.828 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2023-11-30T21:40:12.828 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2023-11-30T21:40:12.828 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /dev 2023-11-30T21:40:12.828 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /run 2023-11-30T21:40:12.828 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2023-11-30T21:40:12.828 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2023-11-30T21:40:12.828 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2023-11-30T21:40:12.828 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2023-11-30T21:40:12.828 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /dev 2023-11-30T21:40:12.828 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:40:13.820 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-30T21:40:13.883 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-30T21:40:13.883 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-30T21:40:14.068 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-30T21:40:14.069 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-30T21:40:14.232 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-30T21:40:14.350 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-30T21:40:14.350 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-30T21:40:14.590 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-30T21:40:14.590 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-30T21:40:14.811 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-30T21:40:14.811 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-30T21:40:14.812 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-30T21:40:14.812 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-30T21:40:14.812 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-30T21:40:14.812 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-30T21:40:14.812 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-30T21:40:14.812 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-30T21:40:14.812 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-30T21:40:14.812 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-30T21:40:14.812 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-30T21:40:14.812 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-30T21:40:14.812 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-30T21:40:14.812 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-30T21:40:14.813 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-30T21:40:14.813 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-30T21:40:14.813 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-30T21:40:14.813 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-30T21:40:14.813 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-30T21:40:14.813 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-30T21:40:14.813 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-30T21:40:14.813 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-30T21:40:14.813 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-30T21:40:14.813 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-30T21:40:14.813 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-30T21:40:14.813 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-30T21:40:14.814 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-30T21:40:14.814 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-30T21:40:14.814 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-30T21:40:15.324 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-30T21:40:15.325 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-30T21:40:15.325 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-30T21:40:15.325 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-30T21:40:15.325 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-30T21:40:15.325 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-30T21:40:15.325 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-30T21:40:15.325 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-30T21:40:15.325 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-30T21:40:15.325 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-30T21:40:15.325 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-30T21:40:15.325 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-30T21:40:15.326 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-30T21:40:15.326 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-30T21:40:15.326 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-30T21:40:15.326 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-30T21:40:15.326 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-30T21:40:15.326 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-30T21:40:15.326 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-30T21:40:15.326 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-30T21:40:15.326 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-30T21:40:15.326 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-30T21:40:15.326 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-30T21:40:15.326 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-30T21:40:15.327 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-30T21:40:15.327 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-30T21:40:15.327 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-30T21:40:15.327 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-30T21:40:15.327 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-30T21:40:16.124 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-30T21:40:16.124 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:16.124 INFO:teuthology.orchestra.run.smithi049.stdout:Upgraded: 2023-11-30T21:40:16.124 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.124 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:40:16.124 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.124 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.124 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.125 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.125 INFO:teuthology.orchestra.run.smithi049.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-30T21:40:16.125 INFO:teuthology.orchestra.run.smithi049.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-30T21:40:16.125 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-30T21:40:16.125 INFO:teuthology.orchestra.run.smithi049.stdout: leveldb-1.22-1.el8.x86_64 2023-11-30T21:40:16.125 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.125 INFO:teuthology.orchestra.run.smithi049.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-30T21:40:16.125 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-30T21:40:16.125 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-30T21:40:16.125 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.125 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-30T21:40:16.126 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.126 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-30T21:40:16.126 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-30T21:40:16.126 INFO:teuthology.orchestra.run.smithi049.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-30T21:40:16.126 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.126 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.126 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.126 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-30T21:40:16.126 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.126 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.126 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.127 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:16.127 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:40:16.356 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-test 2023-11-30T21:40:16.813 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-30T21:40:16.814 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:40:16.814 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2023-11-30T21:40:16.814 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.814 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:40:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-30T21:40:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-30T21:40:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-30T21:40:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: leveldb-1.22-1.el8.x86_64 2023-11-30T21:40:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-30T21:40:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-30T21:40:16.815 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-30T21:40:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-30T21:40:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-30T21:40:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-30T21:40:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-30T21:40:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-30T21:40:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.816 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:16.817 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:40:16.817 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:40:17.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-test 2023-11-30T21:40:17.242 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:44 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:40:17.883 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:46 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:40:19.038 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:40:19.039 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:40:19.039 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-11-30T21:40:19.039 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:40:19.039 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:40:19.039 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-30T21:40:19.039 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-30T21:40:19.040 INFO:teuthology.orchestra.run.smithi049.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-30T21:40:19.040 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-30T21:40:19.040 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-30T21:40:19.040 INFO:teuthology.orchestra.run.smithi049.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-30T21:40:19.040 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-30T21:40:19.040 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:19.040 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:40:19.040 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:40:19.040 INFO:teuthology.orchestra.run.smithi049.stdout:Install 6 Packages 2023-11-30T21:40:19.040 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:19.041 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 53 M 2023-11-30T21:40:19.041 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 229 M 2023-11-30T21:40:19.041 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:40:19.190 INFO:teuthology.orchestra.run.smithi049.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 169 kB 00:00 2023-11-30T21:40:19.675 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:40:19.676 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:40:19.676 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2023-11-30T21:40:19.676 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:40:19.676 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:40:19.676 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-30T21:40:19.676 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-11-30T21:40:19.677 INFO:teuthology.orchestra.run.smithi027.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-30T21:40:19.677 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-30T21:40:19.677 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-30T21:40:19.677 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-30T21:40:19.677 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-30T21:40:19.677 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:40:19.677 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:40:19.677 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:40:19.677 INFO:teuthology.orchestra.run.smithi027.stdout:Install 6 Packages 2023-11-30T21:40:19.677 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:40:19.678 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 53 M 2023-11-30T21:40:19.678 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 229 M 2023-11-30T21:40:19.678 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:40:19.882 INFO:teuthology.orchestra.run.smithi049.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 65 MB/s | 52 MB 00:00 2023-11-30T21:40:19.907 INFO:teuthology.orchestra.run.smithi049.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 248 kB/s | 203 kB 00:00 2023-11-30T21:40:19.932 INFO:teuthology.orchestra.run.smithi027.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.9 MB/s | 169 kB 00:00 2023-11-30T21:40:19.933 INFO:teuthology.orchestra.run.smithi049.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 252 kB/s | 187 kB 00:00 2023-11-30T21:40:20.041 INFO:teuthology.orchestra.run.smithi049.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 652 kB/s | 71 kB 00:00 2023-11-30T21:40:20.408 INFO:teuthology.orchestra.run.smithi049.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 615 kB/s | 323 kB 00:00 2023-11-30T21:40:20.409 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:40:20.409 INFO:teuthology.orchestra.run.smithi049.stdout:Total 39 MB/s | 53 MB 00:01 2023-11-30T21:40:20.455 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:40:20.479 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:40:20.479 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:40:20.666 INFO:teuthology.orchestra.run.smithi027.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 66 MB/s | 52 MB 00:00 2023-11-30T21:40:20.733 INFO:teuthology.orchestra.run.smithi027.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 237 kB/s | 203 kB 00:00 2023-11-30T21:40:20.759 INFO:teuthology.orchestra.run.smithi027.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 227 kB/s | 187 kB 00:00 2023-11-30T21:40:20.808 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:40:20.812 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:40:20.869 INFO:teuthology.orchestra.run.smithi027.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 654 kB/s | 71 kB 00:00 2023-11-30T21:40:21.192 INFO:teuthology.orchestra.run.smithi027.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 615 kB/s | 323 kB 00:00 2023-11-30T21:40:21.193 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:40:21.193 INFO:teuthology.orchestra.run.smithi027.stdout:Total 35 MB/s | 53 MB 00:01 2023-11-30T21:40:21.240 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:40:21.263 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:40:21.263 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:40:21.338 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:40:21.584 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:40:21.587 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-30T21:40:21.587 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:40:21.885 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-30T21:40:22.056 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-30T21:40:22.094 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:40:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-30T21:40:22.323 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-30T21:40:22.351 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-30T21:40:22.448 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-30T21:40:22.574 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-30T21:40:22.787 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-30T21:40:22.873 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-30T21:40:23.070 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-30T21:40:23.253 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-30T21:40:27.618 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-30T21:40:27.813 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-30T21:40:28.144 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-30T21:40:28.144 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-30T21:40:28.144 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-30T21:40:28.144 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-30T21:40:28.144 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-30T21:40:28.144 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-30T21:40:28.447 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-30T21:40:28.693 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-30T21:40:28.892 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-30T21:40:28.892 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:28.892 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:40:28.892 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-30T21:40:28.892 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-30T21:40:28.892 INFO:teuthology.orchestra.run.smithi049.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-30T21:40:28.892 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:28.892 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:40:28.993 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-30T21:40:28.994 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-30T21:40:28.994 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-30T21:40:28.994 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-30T21:40:28.994 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-30T21:40:28.995 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-30T21:40:29.104 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph 2023-11-30T21:40:29.771 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-30T21:40:29.771 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:40:29.771 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:40:29.771 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-30T21:40:29.771 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-30T21:40:29.771 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-30T21:40:29.771 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:40:29.771 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:40:29.991 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:56 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:40:30.042 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph 2023-11-30T21:40:30.936 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:59 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:40:31.804 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:40:31.808 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:40:31.808 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-30T21:40:31.808 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:40:31.808 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:40:31.808 INFO:teuthology.orchestra.run.smithi049.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-30T21:40:31.808 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-30T21:40:31.808 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-30T21:40:31.808 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-30T21:40:31.808 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-30T21:40:31.808 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-30T21:40:31.808 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-30T21:40:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-30T21:40:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-30T21:40:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-30T21:40:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-30T21:40:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-30T21:40:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-30T21:40:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-30T21:40:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-30T21:40:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-30T21:40:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-30T21:40:31.809 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-30T21:40:31.810 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-30T21:40:31.810 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-30T21:40:31.810 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-30T21:40:31.810 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-30T21:40:31.810 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-30T21:40:31.810 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-30T21:40:31.810 INFO:teuthology.orchestra.run.smithi049.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-30T21:40:31.810 INFO:teuthology.orchestra.run.smithi049.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-30T21:40:31.810 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-30T21:40:31.810 INFO:teuthology.orchestra.run.smithi049.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-30T21:40:31.810 INFO:teuthology.orchestra.run.smithi049.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-30T21:40:31.810 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-30T21:40:31.811 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-30T21:40:31.811 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-30T21:40:31.811 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-30T21:40:31.811 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:31.811 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:40:31.811 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:40:31.811 INFO:teuthology.orchestra.run.smithi049.stdout:Install 32 Packages 2023-11-30T21:40:31.811 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:31.812 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 31 M 2023-11-30T21:40:31.812 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 111 M 2023-11-30T21:40:31.812 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:40:32.118 INFO:teuthology.orchestra.run.smithi049.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 128 kB/s | 6.3 kB 00:00 2023-11-30T21:40:32.168 INFO:teuthology.orchestra.run.smithi049.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2023-11-30T21:40:32.227 INFO:teuthology.orchestra.run.smithi049.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2023-11-30T21:40:32.260 INFO:teuthology.orchestra.run.smithi049.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2023-11-30T21:40:32.344 INFO:teuthology.orchestra.run.smithi049.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2023-11-30T21:40:32.620 INFO:teuthology.orchestra.run.smithi049.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 40 MB/s | 18 MB 00:00 2023-11-30T21:40:32.711 INFO:teuthology.orchestra.run.smithi049.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 348 kB/s | 157 kB 00:00 2023-11-30T21:40:32.733 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:40:32.736 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:40:32.736 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-11-30T21:40:32.736 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:40:32.737 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:40:32.737 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-30T21:40:32.737 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-11-30T21:40:32.737 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-30T21:40:32.737 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-30T21:40:32.737 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-30T21:40:32.738 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-30T21:40:32.738 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-30T21:40:32.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-30T21:40:32.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-30T21:40:32.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-30T21:40:32.738 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-30T21:40:32.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-30T21:40:32.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-30T21:40:32.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-30T21:40:32.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-30T21:40:32.739 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-30T21:40:32.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-30T21:40:32.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-30T21:40:32.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-30T21:40:32.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-30T21:40:32.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-30T21:40:32.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-30T21:40:32.740 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-30T21:40:32.741 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-30T21:40:32.741 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-30T21:40:32.741 INFO:teuthology.orchestra.run.smithi027.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-30T21:40:32.741 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-30T21:40:32.741 INFO:teuthology.orchestra.run.smithi027.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-30T21:40:32.741 INFO:teuthology.orchestra.run.smithi027.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-30T21:40:32.741 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-30T21:40:32.742 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-30T21:40:32.742 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-30T21:40:32.742 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-30T21:40:32.742 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:40:32.742 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:40:32.742 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:40:32.743 INFO:teuthology.orchestra.run.smithi027.stdout:Install 32 Packages 2023-11-30T21:40:32.743 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:40:32.744 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 31 M 2023-11-30T21:40:32.744 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 111 M 2023-11-30T21:40:32.744 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:40:32.811 INFO:teuthology.orchestra.run.smithi049.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 83 kB/s | 39 kB 00:00 2023-11-30T21:40:32.837 INFO:teuthology.orchestra.run.smithi049.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 455 kB/s | 57 kB 00:00 2023-11-30T21:40:33.053 INFO:teuthology.orchestra.run.smithi049.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 238 kB/s | 103 kB 00:00 2023-11-30T21:40:33.121 INFO:teuthology.orchestra.run.smithi049.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 877 kB/s | 249 kB 00:00 2023-11-30T21:40:33.195 INFO:teuthology.orchestra.run.smithi049.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.2 MB/s | 457 kB 00:00 2023-11-30T21:40:33.230 INFO:teuthology.orchestra.run.smithi049.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.0 MB/s | 109 kB 00:00 2023-11-30T21:40:33.363 INFO:teuthology.orchestra.run.smithi049.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 2.0 MB/s | 641 kB 00:00 2023-11-30T21:40:33.369 INFO:teuthology.orchestra.run.smithi027.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 93 kB/s | 6.3 kB 00:00 2023-11-30T21:40:33.420 INFO:teuthology.orchestra.run.smithi027.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2023-11-30T21:40:33.470 INFO:teuthology.orchestra.run.smithi027.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2023-11-30T21:40:33.504 INFO:teuthology.orchestra.run.smithi027.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2023-11-30T21:40:33.613 INFO:teuthology.orchestra.run.smithi027.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2023-11-30T21:40:33.705 INFO:teuthology.orchestra.run.smithi049.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 86 kB/s | 44 kB 00:00 2023-11-30T21:40:33.847 INFO:teuthology.orchestra.run.smithi049.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 300 kB/s | 185 kB 00:00 2023-11-30T21:40:33.896 INFO:teuthology.orchestra.run.smithi027.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 38 MB/s | 18 MB 00:00 2023-11-30T21:40:33.946 INFO:teuthology.orchestra.run.smithi027.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 355 kB/s | 157 kB 00:00 2023-11-30T21:40:33.947 INFO:teuthology.orchestra.run.smithi049.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2023-11-30T21:40:33.973 INFO:teuthology.orchestra.run.smithi049.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 284 kB/s | 173 kB 00:00 2023-11-30T21:40:33.998 INFO:teuthology.orchestra.run.smithi049.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.3 MB/s | 384 kB 00:00 2023-11-30T21:40:34.038 INFO:teuthology.orchestra.run.smithi027.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 622 kB/s | 57 kB 00:00 2023-11-30T21:40:34.039 INFO:teuthology.orchestra.run.smithi049.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 191 kB/s | 18 kB 00:00 2023-11-30T21:40:34.072 INFO:teuthology.orchestra.run.smithi027.stdout:(9/32): python3-markupsafe-0.23-19.el8.x86_64.r 84 kB/s | 39 kB 00:00 2023-11-30T21:40:34.073 INFO:teuthology.orchestra.run.smithi049.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 485 kB/s | 49 kB 00:00 2023-11-30T21:40:34.090 INFO:teuthology.orchestra.run.smithi049.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 638 kB/s | 59 kB 00:00 2023-11-30T21:40:34.174 INFO:teuthology.orchestra.run.smithi049.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 155 kB/s | 16 kB 00:00 2023-11-30T21:40:34.199 INFO:teuthology.orchestra.run.smithi049.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 172 kB/s | 19 kB 00:00 2023-11-30T21:40:34.231 INFO:teuthology.orchestra.run.smithi027.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.3 MB/s | 457 kB 00:00 2023-11-30T21:40:34.233 INFO:teuthology.orchestra.run.smithi049.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-11-30T21:40:34.275 INFO:teuthology.orchestra.run.smithi049.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2023-11-30T21:40:34.300 INFO:teuthology.orchestra.run.smithi049.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2023-11-30T21:40:34.349 INFO:teuthology.orchestra.run.smithi027.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 229 kB/s | 103 kB 00:00 2023-11-30T21:40:34.375 INFO:teuthology.orchestra.run.smithi049.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 194 kB/s | 27 kB 00:00 2023-11-30T21:40:34.390 INFO:teuthology.orchestra.run.smithi027.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 784 kB/s | 249 kB 00:00 2023-11-30T21:40:34.409 INFO:teuthology.orchestra.run.smithi049.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.3 MB/s | 251 kB 00:00 2023-11-30T21:40:34.459 INFO:teuthology.orchestra.run.smithi049.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.4 MB/s | 255 kB 00:00 2023-11-30T21:40:34.484 INFO:teuthology.orchestra.run.smithi049.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 783 kB/s | 85 kB 00:00 2023-11-30T21:40:34.509 INFO:teuthology.orchestra.run.smithi049.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2023-11-30T21:40:34.514 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:40:34.515 INFO:teuthology.orchestra.run.smithi049.stdout:Total 11 MB/s | 31 MB 00:02 2023-11-30T21:40:34.572 INFO:teuthology.orchestra.run.smithi027.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 1.8 MB/s | 641 kB 00:00 2023-11-30T21:40:34.598 INFO:teuthology.orchestra.run.smithi027.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 437 kB/s | 109 kB 00:00 2023-11-30T21:40:34.600 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:40:34.643 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:40:34.643 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:40:34.817 INFO:teuthology.orchestra.run.smithi027.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 102 kB/s | 44 kB 00:00 2023-11-30T21:40:34.957 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:40:34.960 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:40:35.282 INFO:teuthology.orchestra.run.smithi027.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 261 kB/s | 185 kB 00:00 2023-11-30T21:40:35.399 INFO:teuthology.orchestra.run.smithi027.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 216 kB/s | 173 kB 00:00 2023-11-30T21:40:35.425 INFO:teuthology.orchestra.run.smithi027.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 78 kB/s | 11 kB 00:00 2023-11-30T21:40:35.516 INFO:teuthology.orchestra.run.smithi027.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 549 kB/s | 384 kB 00:00 2023-11-30T21:40:35.542 INFO:teuthology.orchestra.run.smithi027.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 150 kB/s | 18 kB 00:00 2023-11-30T21:40:35.567 INFO:teuthology.orchestra.run.smithi027.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 343 kB/s | 49 kB 00:00 2023-11-30T21:40:35.634 INFO:teuthology.orchestra.run.smithi027.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 501 kB/s | 59 kB 00:00 2023-11-30T21:40:35.726 INFO:teuthology.orchestra.run.smithi027.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 98 kB/s | 16 kB 00:00 2023-11-30T21:40:35.768 INFO:teuthology.orchestra.run.smithi027.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 140 kB/s | 19 kB 00:00 2023-11-30T21:40:35.868 INFO:teuthology.orchestra.run.smithi027.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 168 kB/s | 24 kB 00:00 2023-11-30T21:40:35.955 INFO:teuthology.orchestra.run.smithi027.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 159 kB/s | 29 kB 00:00 2023-11-30T21:40:36.019 INFO:teuthology.orchestra.run.smithi027.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 183 kB/s | 27 kB 00:00 2023-11-30T21:40:36.103 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:40:36.169 INFO:teuthology.orchestra.run.smithi027.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 451 kB/s | 283 kB 00:00 2023-11-30T21:40:36.297 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-30T21:40:36.328 INFO:teuthology.orchestra.run.smithi027.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 539 kB/s | 85 kB 00:00 2023-11-30T21:40:36.386 INFO:teuthology.orchestra.run.smithi027.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 588 kB/s | 255 kB 00:00 2023-11-30T21:40:36.445 INFO:teuthology.orchestra.run.smithi027.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 197 kB/s | 23 kB 00:00 2023-11-30T21:40:36.454 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-30T21:40:36.520 INFO:teuthology.orchestra.run.smithi027.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 501 kB/s | 251 kB 00:00 2023-11-30T21:40:36.522 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:40:36.522 INFO:teuthology.orchestra.run.smithi027.stdout:Total 8.2 MB/s | 31 MB 00:03 2023-11-30T21:40:36.608 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:40:36.650 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:40:36.650 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:40:36.668 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-30T21:40:36.836 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-30T21:40:36.978 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:40:36.982 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:40:36.991 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-30T21:40:37.165 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-30T21:40:37.341 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-30T21:40:37.502 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-30T21:40:37.680 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-30T21:40:37.909 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-30T21:40:38.088 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-30T21:40:38.284 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:40:38.297 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-30T21:40:38.519 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-30T21:40:38.717 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-30T21:40:38.743 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-30T21:40:39.008 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-30T21:40:39.023 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-30T21:40:39.153 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-30T21:40:39.241 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-30T21:40:39.381 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-30T21:40:39.447 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-30T21:40:39.580 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-30T21:40:39.656 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-30T21:40:39.852 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-30T21:40:39.871 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-30T21:40:40.016 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-30T21:40:40.043 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-30T21:40:40.239 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-30T21:40:40.244 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-30T21:40:40.422 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-30T21:40:40.489 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-30T21:40:40.656 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-30T21:40:40.718 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-30T21:40:40.882 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-30T21:40:41.026 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-30T21:40:41.110 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-30T21:40:41.441 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-30T21:40:41.461 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-30T21:40:41.716 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-30T21:40:41.755 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-30T21:40:41.951 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-30T21:40:42.130 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-30T21:40:42.212 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-30T21:40:42.427 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-30T21:40:42.432 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-30T21:40:42.784 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-30T21:40:42.999 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-30T21:40:43.196 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-30T21:40:43.599 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-30T21:40:43.860 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-30T21:40:44.069 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-30T21:40:44.139 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-30T21:40:44.265 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-30T21:40:44.397 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-30T21:40:44.726 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-30T21:40:44.756 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-30T21:40:44.956 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-30T21:40:45.098 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-30T21:40:45.195 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-30T21:40:45.353 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-30T21:40:45.371 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-30T21:40:45.533 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-30T21:40:45.536 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-30T21:40:45.769 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-30T21:40:45.968 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-30T21:40:45.969 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-30T21:40:45.969 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-30T21:40:45.969 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-30T21:40:45.969 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-30T21:40:45.969 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-30T21:40:45.969 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-30T21:40:45.969 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-30T21:40:45.969 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-30T21:40:45.969 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-30T21:40:45.969 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-30T21:40:45.969 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-30T21:40:45.970 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-30T21:40:45.970 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-30T21:40:45.970 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-30T21:40:45.970 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-30T21:40:45.970 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-30T21:40:45.970 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-30T21:40:45.970 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-30T21:40:45.970 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-30T21:40:45.970 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-30T21:40:45.970 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-30T21:40:45.970 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-30T21:40:45.970 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-30T21:40:45.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-30T21:40:45.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-30T21:40:45.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-30T21:40:45.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-30T21:40:45.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-30T21:40:45.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-30T21:40:45.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-30T21:40:45.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-30T21:40:47.108 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-30T21:40:47.109 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:47.109 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:40:47.109 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:47.109 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:47.109 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:47.109 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-30T21:40:47.109 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:47.109 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:47.109 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-30T21:40:47.109 INFO:teuthology.orchestra.run.smithi049.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-30T21:40:47.109 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-30T21:40:47.110 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-30T21:40:47.110 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-30T21:40:47.110 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-30T21:40:47.110 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-30T21:40:47.110 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-30T21:40:47.110 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-30T21:40:47.110 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-30T21:40:47.110 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-30T21:40:47.110 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-30T21:40:47.110 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-30T21:40:47.110 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend-2.6-1.el8.noarch 2023-11-30T21:40:47.111 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-30T21:40:47.111 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-30T21:40:47.111 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-30T21:40:47.111 INFO:teuthology.orchestra.run.smithi049.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-30T21:40:47.111 INFO:teuthology.orchestra.run.smithi049.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-30T21:40:47.111 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-30T21:40:47.111 INFO:teuthology.orchestra.run.smithi049.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-30T21:40:47.111 INFO:teuthology.orchestra.run.smithi049.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-30T21:40:47.111 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-30T21:40:47.111 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-30T21:40:47.111 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-30T21:40:47.112 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-30T21:40:47.112 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:47.112 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:40:47.470 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-30T21:40:47.869 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-30T21:40:47.889 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-base 2023-11-30T21:40:48.331 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-30T21:40:48.619 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-30T21:40:48.790 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:15 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:40:48.854 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-30T21:40:49.201 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-30T21:40:49.220 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-30T21:40:49.409 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-30T21:40:49.846 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-30T21:40:49.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-30T21:40:49.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-30T21:40:49.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-30T21:40:49.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-30T21:40:49.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-30T21:40:49.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-30T21:40:49.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-30T21:40:49.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-30T21:40:49.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-30T21:40:49.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-30T21:40:49.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-30T21:40:49.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-30T21:40:49.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-30T21:40:49.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-30T21:40:49.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-30T21:40:49.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-30T21:40:49.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-30T21:40:49.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-30T21:40:49.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-30T21:40:49.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-30T21:40:49.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-30T21:40:49.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-30T21:40:49.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-30T21:40:49.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-30T21:40:49.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-30T21:40:49.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-30T21:40:49.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-30T21:40:49.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-30T21:40:49.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-30T21:40:49.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-30T21:40:49.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-30T21:40:50.323 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:40:50.561 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:40:50.565 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-30T21:40:50.566 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:40:50.679 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install cephadm 2023-11-30T21:40:51.187 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-30T21:40:51.187 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:40:51.188 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:40:51.188 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:51.188 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:51.189 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:51.189 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-30T21:40:51.190 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:51.190 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-30T21:40:51.190 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-30T21:40:51.191 INFO:teuthology.orchestra.run.smithi027.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-30T21:40:51.191 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-30T21:40:51.192 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-30T21:40:51.193 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-30T21:40:51.193 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-30T21:40:51.194 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-30T21:40:51.195 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-30T21:40:51.196 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-30T21:40:51.196 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-30T21:40:51.197 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-30T21:40:51.197 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-30T21:40:51.198 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-30T21:40:51.201 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-2.6-1.el8.noarch 2023-11-30T21:40:51.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-30T21:40:51.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-30T21:40:51.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-30T21:40:51.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-30T21:40:51.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-30T21:40:51.202 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-30T21:40:51.203 INFO:teuthology.orchestra.run.smithi027.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-30T21:40:51.203 INFO:teuthology.orchestra.run.smithi027.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-30T21:40:51.204 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-30T21:40:51.204 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-30T21:40:51.204 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-30T21:40:51.204 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-30T21:40:51.205 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:40:51.205 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:40:51.531 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-base 2023-11-30T21:40:51.586 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:18 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:40:52.428 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:21 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:40:53.394 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:40:53.394 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:40:53.395 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-11-30T21:40:53.395 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:40:53.395 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:40:53.395 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-30T21:40:53.395 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:53.395 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:40:53.395 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:40:53.395 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-30T21:40:53.395 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:53.396 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 71 k 2023-11-30T21:40:53.396 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 290 k 2023-11-30T21:40:53.396 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:40:53.431 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm-16.2.4-0.el8.noarch.rpm 2.0 MB/s | 71 kB 00:00 2023-11-30T21:40:53.432 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:40:53.432 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.9 MB/s | 71 kB 00:00 2023-11-30T21:40:53.433 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:40:53.440 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:40:53.440 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:40:53.442 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:40:53.445 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:40:53.487 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:40:53.970 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:40:54.141 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-30T21:40:54.209 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:40:54.211 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-11-30T21:40:54.211 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:40:54.328 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install cephadm 2023-11-30T21:40:54.360 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-30T21:40:54.595 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-30T21:40:55.219 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:24 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:40:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-30T21:40:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:55.291 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:40:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-30T21:40:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:55.292 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:40:55.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-immutable-object-cache 2023-11-30T21:40:56.451 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:23 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:40:57.076 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:40:57.077 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:40:57.077 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2023-11-30T21:40:57.077 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:40:57.077 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:40:57.077 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-30T21:40:57.077 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:40:57.077 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:40:57.077 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:40:57.077 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-11-30T21:40:57.077 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:40:57.078 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 71 k 2023-11-30T21:40:57.079 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 290 k 2023-11-30T21:40:57.079 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:40:57.129 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 71 kB 00:00 2023-11-30T21:40:57.130 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:40:57.130 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.3 MB/s | 71 kB 00:00 2023-11-30T21:40:57.130 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:40:57.138 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:40:57.138 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:40:57.142 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:40:57.143 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:40:57.188 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:40:57.824 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-30T21:40:58.243 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:40:58.244 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:40:58.244 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2023-11-30T21:40:58.244 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:40:58.244 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:40:58.244 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-30T21:40:58.244 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:58.244 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:40:58.244 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:40:58.244 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-30T21:40:58.244 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:40:58.245 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 163 k 2023-11-30T21:40:58.245 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 587 k 2023-11-30T21:40:58.245 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:40:58.704 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 353 kB/s | 163 kB 00:00 2023-11-30T21:40:58.705 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:40:58.705 INFO:teuthology.orchestra.run.smithi049.stdout:Total 352 kB/s | 163 kB 00:00 2023-11-30T21:40:58.706 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:40:58.719 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:40:58.719 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:40:58.776 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:40:58.779 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:40:59.522 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:40:59.750 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:40:59.982 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-30T21:41:00.178 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:41:00.227 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-30T21:41:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:41:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:00.933 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:41:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-30T21:41:00.933 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:00.934 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:41:01.089 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-30T21:41:01.089 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:01.089 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:41:01.089 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-30T21:41:01.089 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:01.089 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:41:01.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr 2023-11-30T21:41:01.346 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-immutable-object-cache 2023-11-30T21:41:02.047 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:29 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:41:02.245 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:31 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:41:03.609 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:41:03.850 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:41:03.851 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-30T21:41:03.852 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:41:03.961 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-dashboard 2023-11-30T21:41:04.036 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:41:04.036 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:04.037 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2023-11-30T21:41:04.037 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:04.037 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:41:04.037 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-30T21:41:04.037 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:04.037 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:41:04.037 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:04.037 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-11-30T21:41:04.037 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:04.037 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 163 k 2023-11-30T21:41:04.037 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 587 k 2023-11-30T21:41:04.037 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:41:04.086 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.4 MB/s | 163 kB 00:00 2023-11-30T21:41:04.086 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:41:04.086 INFO:teuthology.orchestra.run.smithi027.stdout:Total 3.3 MB/s | 163 kB 00:00 2023-11-30T21:41:04.086 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:41:04.098 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:41:04.098 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:41:04.149 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:41:04.152 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:41:04.271 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:41:04.498 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:41:04.856 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:31 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:41:04.975 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:41:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:41:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:05.772 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:41:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-30T21:41:05.772 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:05.772 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:41:06.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr 2023-11-30T21:41:06.652 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:41:06.654 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:06.654 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-30T21:41:06.654 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:06.654 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:41:06.654 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-30T21:41:06.654 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-30T21:41:06.654 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-30T21:41:06.654 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-30T21:41:06.654 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-30T21:41:06.654 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-30T21:41:06.655 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-30T21:41:06.655 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:06.655 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:41:06.655 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:06.655 INFO:teuthology.orchestra.run.smithi049.stdout:Install 6 Packages 2023-11-30T21:41:06.655 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:06.655 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 3.0 M 2023-11-30T21:41:06.656 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 15 M 2023-11-30T21:41:06.656 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:41:07.038 INFO:teuthology.orchestra.run.smithi049.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 437 kB/s | 24 kB 00:00 2023-11-30T21:41:07.072 INFO:teuthology.orchestra.run.smithi049.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 100 kB/s | 8.9 kB 00:00 2023-11-30T21:41:07.085 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:36 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:41:07.165 INFO:teuthology.orchestra.run.smithi049.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 15 MB/s | 2.7 MB 00:00 2023-11-30T21:41:07.306 INFO:teuthology.orchestra.run.smithi049.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 144 kB/s | 34 kB 00:00 2023-11-30T21:41:07.331 INFO:teuthology.orchestra.run.smithi049.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 145 kB/s | 43 kB 00:00 2023-11-30T21:41:07.457 INFO:teuthology.orchestra.run.smithi049.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 685 kB/s | 196 kB 00:00 2023-11-30T21:41:07.460 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:41:07.460 INFO:teuthology.orchestra.run.smithi049.stdout:Total 3.7 MB/s | 3.0 MB 00:00 2023-11-30T21:41:07.472 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:41:07.482 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:41:07.482 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:41:07.661 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:41:07.664 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:41:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:41:08.183 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-30T21:41:08.476 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-30T21:41:08.640 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:41:08.659 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-30T21:41:08.772 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-30T21:41:08.881 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:41:08.882 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-11-30T21:41:08.882 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:41:09.009 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-dashboard 2023-11-30T21:41:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-30T21:41:09.904 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:38 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:41:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-30T21:41:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-30T21:41:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-30T21:41:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-30T21:41:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-30T21:41:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-30T21:41:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-30T21:41:11.290 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-30T21:41:11.290 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:11.290 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:41:11.290 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-30T21:41:11.290 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-30T21:41:11.290 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-30T21:41:11.290 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-30T21:41:11.291 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-30T21:41:11.291 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-30T21:41:11.291 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:11.291 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:41:11.423 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-30T21:41:11.714 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:41:11.715 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:11.715 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-11-30T21:41:11.715 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:11.717 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:41:11.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-30T21:41:11.717 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-11-30T21:41:11.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-30T21:41:11.717 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-30T21:41:11.717 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-30T21:41:11.717 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-30T21:41:11.718 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-30T21:41:11.718 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:11.718 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:41:11.718 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:11.718 INFO:teuthology.orchestra.run.smithi027.stdout:Install 6 Packages 2023-11-30T21:41:11.718 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:11.718 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 3.0 M 2023-11-30T21:41:11.719 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 15 M 2023-11-30T21:41:11.719 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:41:12.173 INFO:teuthology.orchestra.run.smithi027.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 520 kB/s | 24 kB 00:00 2023-11-30T21:41:12.190 INFO:teuthology.orchestra.run.smithi027.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 141 kB/s | 8.9 kB 00:00 2023-11-30T21:41:12.249 INFO:teuthology.orchestra.run.smithi027.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 22 MB/s | 2.7 MB 00:00 2023-11-30T21:41:12.308 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:39 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:41:12.407 INFO:teuthology.orchestra.run.smithi027.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 182 kB/s | 43 kB 00:00 2023-11-30T21:41:12.550 INFO:teuthology.orchestra.run.smithi027.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 94 kB/s | 34 kB 00:00 2023-11-30T21:41:12.774 INFO:teuthology.orchestra.run.smithi027.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 374 kB/s | 196 kB 00:00 2023-11-30T21:41:12.777 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:41:12.777 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.8 MB/s | 3.0 MB 00:01 2023-11-30T21:41:12.789 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:41:12.801 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:41:12.801 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:41:12.999 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:41:13.002 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:41:13.322 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:41:13.597 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-30T21:41:13.992 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-30T21:41:14.102 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:41:14.103 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================================ 2023-11-30T21:41:14.104 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-30T21:41:14.104 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================================ 2023-11-30T21:41:14.104 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:41:14.104 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-30T21:41:14.104 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-30T21:41:14.104 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-30T21:41:14.104 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-30T21:41:14.104 INFO:teuthology.orchestra.run.smithi049.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-30T21:41:14.104 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-30T21:41:14.104 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-30T21:41:14.104 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-30T21:41:14.105 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-30T21:41:14.105 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:14.105 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:41:14.105 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================================ 2023-11-30T21:41:14.105 INFO:teuthology.orchestra.run.smithi049.stdout:Install 8 Packages 2023-11-30T21:41:14.105 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:14.106 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 37 M 2023-11-30T21:41:14.106 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 210 M 2023-11-30T21:41:14.106 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:41:14.216 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-30T21:41:14.379 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-30T21:41:14.504 INFO:teuthology.orchestra.run.smithi049.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 52 MB/s | 8.6 MB 00:00 2023-11-30T21:41:15.395 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-30T21:41:15.873 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-30T21:41:16.165 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-30T21:41:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-30T21:41:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-30T21:41:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-30T21:41:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-30T21:41:16.166 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-30T21:41:16.206 INFO:teuthology.orchestra.run.smithi049.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 2.2 MB/s | 3.7 MB 00:01 2023-11-30T21:41:16.581 INFO:teuthology.orchestra.run.smithi049.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 2.2 MB/s | 4.9 MB 00:02 2023-11-30T21:41:16.840 INFO:teuthology.orchestra.run.smithi049.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 1.9 MB/s | 4.7 MB 00:02 2023-11-30T21:41:16.865 INFO:teuthology.orchestra.run.smithi049.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 341 kB/s | 225 kB 00:00 2023-11-30T21:41:17.091 INFO:teuthology.orchestra.run.smithi049.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 767 kB/s | 172 kB 00:00 2023-11-30T21:41:17.165 INFO:teuthology.orchestra.run.smithi049.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.9 MB/s | 645 kB 00:00 2023-11-30T21:41:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-30T21:41:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:17.456 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:41:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-30T21:41:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-30T21:41:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-30T21:41:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-30T21:41:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-30T21:41:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-30T21:41:17.456 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:17.457 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:41:17.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-30T21:41:18.693 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:47 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:41:19.082 INFO:teuthology.orchestra.run.smithi049.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 5.5 MB/s | 14 MB 00:02 2023-11-30T21:41:19.083 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:41:19.083 INFO:teuthology.orchestra.run.smithi049.stdout:Total 7.4 MB/s | 37 MB 00:04 2023-11-30T21:41:19.325 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:41:19.344 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:41:19.344 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:41:19.696 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:41:19.699 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:41:20.320 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:41:20.446 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-30T21:41:20.507 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:41:20.508 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================================ 2023-11-30T21:41:20.508 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-11-30T21:41:20.508 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================================ 2023-11-30T21:41:20.508 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:41:20.508 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-30T21:41:20.508 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-11-30T21:41:20.508 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-30T21:41:20.508 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-30T21:41:20.508 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-30T21:41:20.509 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-30T21:41:20.509 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-30T21:41:20.509 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-30T21:41:20.509 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-30T21:41:20.509 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:20.509 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:41:20.509 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================================ 2023-11-30T21:41:20.509 INFO:teuthology.orchestra.run.smithi027.stdout:Install 8 Packages 2023-11-30T21:41:20.509 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:20.510 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 37 M 2023-11-30T21:41:20.510 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 210 M 2023-11-30T21:41:20.510 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:41:20.566 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-30T21:41:20.746 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-30T21:41:20.932 INFO:teuthology.orchestra.run.smithi027.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 45 MB/s | 8.6 MB 00:00 2023-11-30T21:41:21.382 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-30T21:41:21.546 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-30T21:41:22.106 INFO:teuthology.orchestra.run.smithi027.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.2 MB/s | 3.7 MB 00:01 2023-11-30T21:41:22.194 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-30T21:41:22.290 INFO:teuthology.orchestra.run.smithi027.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 3.0 MB/s | 4.7 MB 00:01 2023-11-30T21:41:22.315 INFO:teuthology.orchestra.run.smithi027.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.1 MB/s | 225 kB 00:00 2023-11-30T21:41:22.338 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-30T21:41:22.382 INFO:teuthology.orchestra.run.smithi027.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 3.0 MB/s | 4.9 MB 00:01 2023-11-30T21:41:22.808 INFO:teuthology.orchestra.run.smithi027.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 406 kB/s | 172 kB 00:00 2023-11-30T21:41:22.858 INFO:teuthology.orchestra.run.smithi027.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2023-11-30T21:41:23.162 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-30T21:41:23.481 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-30T21:41:23.729 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-30T21:41:24.217 INFO:teuthology.orchestra.run.smithi027.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 7.1 MB/s | 14 MB 00:01 2023-11-30T21:41:24.217 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:41:24.218 INFO:teuthology.orchestra.run.smithi027.stdout:Total 9.9 MB/s | 37 MB 00:03 2023-11-30T21:41:24.462 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:41:24.481 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:41:24.482 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:41:24.827 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:41:24.831 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:41:25.549 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:41:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-30T21:41:25.761 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-30T21:41:25.795 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-30T21:41:25.945 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-30T21:41:26.618 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-30T21:41:26.828 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-30T21:41:27.241 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-30T21:41:27.436 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-30T21:41:27.522 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-30T21:41:27.720 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-30T21:41:27.773 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-30T21:41:27.773 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-30T21:41:27.773 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-30T21:41:27.773 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-30T21:41:27.773 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-30T21:41:27.774 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-30T21:41:27.774 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-30T21:41:27.774 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-30T21:41:28.503 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-30T21:41:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-30T21:41:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:28.522 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:41:28.522 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-30T21:41:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-30T21:41:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-30T21:41:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-30T21:41:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-30T21:41:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-30T21:41:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-30T21:41:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-30T21:41:28.523 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:28.523 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:41:28.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-rook 2023-11-30T21:41:28.861 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-30T21:41:29.170 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-30T21:41:29.526 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:56 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:41:31.066 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-30T21:41:31.317 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:41:31.318 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:31.319 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-30T21:41:31.319 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:31.319 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:41:31.319 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-30T21:41:31.319 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-30T21:41:31.319 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-30T21:41:31.319 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-30T21:41:31.319 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-30T21:41:31.319 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-30T21:41:31.319 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-30T21:41:31.319 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-30T21:41:31.319 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-30T21:41:31.320 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-30T21:41:31.320 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-30T21:41:31.320 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-30T21:41:31.320 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-30T21:41:31.320 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-30T21:41:31.320 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:31.320 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:41:31.320 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:31.320 INFO:teuthology.orchestra.run.smithi049.stdout:Install 13 Packages 2023-11-30T21:41:31.320 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:31.321 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 1.9 M 2023-11-30T21:41:31.321 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 24 M 2023-11-30T21:41:31.321 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:41:31.889 INFO:teuthology.orchestra.run.smithi049.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 518 kB/s | 27 kB 00:00 2023-11-30T21:41:32.097 INFO:teuthology.orchestra.run.smithi049.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 77 kB/s | 20 kB 00:00 2023-11-30T21:41:32.137 INFO:teuthology.orchestra.run.smithi049.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 89 kB/s | 27 kB 00:00 2023-11-30T21:41:32.298 INFO:teuthology.orchestra.run.smithi049.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 309 kB/s | 126 kB 00:00 2023-11-30T21:41:32.357 INFO:teuthology.orchestra.run.smithi049.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 425 kB/s | 110 kB 00:00 2023-11-30T21:41:32.382 INFO:teuthology.orchestra.run.smithi049.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 632 kB/s | 155 kB 00:00 2023-11-30T21:41:32.431 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-30T21:41:32.490 INFO:teuthology.orchestra.run.smithi049.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 222 kB/s | 43 kB 00:00 2023-11-30T21:41:32.507 INFO:teuthology.orchestra.run.smithi049.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 130 kB/s | 16 kB 00:00 2023-11-30T21:41:32.524 INFO:teuthology.orchestra.run.smithi049.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 197 kB/s | 33 kB 00:00 2023-11-30T21:41:32.583 INFO:teuthology.orchestra.run.smithi049.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 1.2 MB/s | 64 kB 00:00 2023-11-30T21:41:32.616 INFO:teuthology.orchestra.run.smithi049.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.8 MB/s | 61 kB 00:00 2023-11-30T21:41:32.658 INFO:teuthology.orchestra.run.smithi049.stdout:(12/13): python3-google-auth-1.1.1-10.el8.noarc 485 kB/s | 81 kB 00:00 2023-11-30T21:41:32.819 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-30T21:41:32.842 INFO:teuthology.orchestra.run.smithi049.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.5 MB/s | 1.2 MB 00:00 2023-11-30T21:41:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:41:32.845 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2023-11-30T21:41:32.908 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:41:32.925 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:41:32.925 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:41:33.020 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:41:33.023 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:41:33.172 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-30T21:41:33.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-30T21:41:33.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-30T21:41:33.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-30T21:41:33.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-30T21:41:33.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-30T21:41:33.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-30T21:41:33.173 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-30T21:41:33.569 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:41:33.792 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-30T21:41:33.986 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-30T21:41:33.986 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:33.986 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:41:33.986 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-30T21:41:33.986 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-30T21:41:33.986 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-30T21:41:33.986 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-30T21:41:33.986 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-30T21:41:33.986 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-30T21:41:33.987 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-30T21:41:33.987 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-30T21:41:33.987 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:33.987 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:41:34.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-rook 2023-11-30T21:41:34.097 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-30T21:41:34.315 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-30T21:41:34.577 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-30T21:41:34.782 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-30T21:41:34.985 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:03 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:41:35.005 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-30T21:41:35.323 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-30T21:41:35.527 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-30T21:41:36.644 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-30T21:41:36.791 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:41:36.793 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-11-30T21:41:36.793 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:36.793 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:41:36.793 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-30T21:41:36.794 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-11-30T21:41:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-30T21:41:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-30T21:41:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-30T21:41:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-30T21:41:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-30T21:41:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-30T21:41:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-30T21:41:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-30T21:41:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-30T21:41:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-30T21:41:36.794 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-30T21:41:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-30T21:41:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:36.795 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:41:36.795 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:36.795 INFO:teuthology.orchestra.run.smithi027.stdout:Install 13 Packages 2023-11-30T21:41:36.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:36.795 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 1.9 M 2023-11-30T21:41:36.796 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 24 M 2023-11-30T21:41:36.796 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:41:36.926 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-30T21:41:37.109 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-30T21:41:37.273 INFO:teuthology.orchestra.run.smithi027.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 508 kB/s | 27 kB 00:00 2023-11-30T21:41:37.482 INFO:teuthology.orchestra.run.smithi027.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 76 kB/s | 20 kB 00:00 2023-11-30T21:41:37.498 INFO:teuthology.orchestra.run.smithi027.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 95 kB/s | 27 kB 00:00 2023-11-30T21:41:37.626 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-30T21:41:37.691 INFO:teuthology.orchestra.run.smithi027.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 303 kB/s | 126 kB 00:00 2023-11-30T21:41:37.716 INFO:teuthology.orchestra.run.smithi027.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 714 kB/s | 155 kB 00:00 2023-11-30T21:41:37.741 INFO:teuthology.orchestra.run.smithi027.stdout:(6/13): python3-pyasn1-modules-0.3.7-6.el8.noar 424 kB/s | 110 kB 00:00 2023-11-30T21:41:37.785 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-30T21:41:37.858 INFO:teuthology.orchestra.run.smithi027.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 232 kB/s | 33 kB 00:00 2023-11-30T21:41:37.883 INFO:teuthology.orchestra.run.smithi027.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 115 kB/s | 16 kB 00:00 2023-11-30T21:41:37.909 INFO:teuthology.orchestra.run.smithi027.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 196 kB/s | 43 kB 00:00 2023-11-30T21:41:37.934 INFO:teuthology.orchestra.run.smithi027.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 1.0 MB/s | 81 kB 00:00 2023-11-30T21:41:38.012 INFO:teuthology.orchestra.run.smithi027.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 786 kB/s | 61 kB 00:00 2023-11-30T21:41:38.043 INFO:teuthology.orchestra.run.smithi027.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 480 kB/s | 64 kB 00:00 2023-11-30T21:41:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-30T21:41:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-30T21:41:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-30T21:41:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-30T21:41:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-30T21:41:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-30T21:41:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-30T21:41:38.095 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-30T21:41:38.096 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-30T21:41:38.096 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-30T21:41:38.096 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-30T21:41:38.096 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-30T21:41:38.096 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-30T21:41:38.101 INFO:teuthology.orchestra.run.smithi027.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.3 MB/s | 1.2 MB 00:00 2023-11-30T21:41:38.102 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:41:38.102 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.5 MB/s | 1.9 MB 00:01 2023-11-30T21:41:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:41:38.180 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:41:38.180 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:41:38.274 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:41:38.277 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:41:38.906 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-30T21:41:38.906 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:38.906 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:41:38.906 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-30T21:41:38.906 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-30T21:41:38.906 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-30T21:41:38.907 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-30T21:41:38.907 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-30T21:41:38.907 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-30T21:41:38.907 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-30T21:41:38.907 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-30T21:41:38.907 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-30T21:41:38.907 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-30T21:41:38.907 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-30T21:41:38.908 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-30T21:41:38.908 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-30T21:41:38.908 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:38.908 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:41:39.036 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-cephadm 2023-11-30T21:41:39.280 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:41:39.557 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-30T21:41:39.797 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-30T21:41:39.934 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:06 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:41:40.225 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-30T21:41:40.493 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-30T21:41:40.730 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-30T21:41:41.012 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-30T21:41:41.316 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-30T21:41:41.592 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-30T21:41:41.743 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:41:41.744 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2023-11-30T21:41:41.744 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:41.744 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:41:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-30T21:41:41.744 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-30T21:41:41.744 INFO:teuthology.orchestra.run.smithi049.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-30T21:41:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-30T21:41:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-30T21:41:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-30T21:41:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-30T21:41:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:41.745 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:41:41.745 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:41.745 INFO:teuthology.orchestra.run.smithi049.stdout:Install 6 Packages 2023-11-30T21:41:41.745 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:41.746 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 5.8 M 2023-11-30T21:41:41.746 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 24 M 2023-11-30T21:41:41.746 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:41:42.326 INFO:teuthology.orchestra.run.smithi049.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.9 MB/s | 89 kB 00:00 2023-11-30T21:41:42.601 INFO:teuthology.orchestra.run.smithi049.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 77 kB/s | 21 kB 00:00 2023-11-30T21:41:42.729 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-30T21:41:42.895 INFO:teuthology.orchestra.run.smithi049.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 873 kB/s | 538 kB 00:00 2023-11-30T21:41:43.034 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-30T21:41:43.129 INFO:teuthology.orchestra.run.smithi049.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 674 kB/s | 355 kB 00:00 2023-11-30T21:41:43.146 INFO:teuthology.orchestra.run.smithi049.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 195 kB/s | 49 kB 00:00 2023-11-30T21:41:43.259 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-30T21:41:43.468 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-30T21:41:43.588 INFO:teuthology.orchestra.run.smithi049.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 3.7 MB/s | 4.8 MB 00:01 2023-11-30T21:41:43.589 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:41:43.589 INFO:teuthology.orchestra.run.smithi049.stdout:Total 3.2 MB/s | 5.8 MB 00:01 2023-11-30T21:41:43.645 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:41:43.651 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-30T21:41:43.655 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:41:43.656 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:41:43.769 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:41:43.771 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:41:43.964 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-30T21:41:43.964 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-30T21:41:43.964 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-30T21:41:43.964 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-30T21:41:43.965 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-30T21:41:43.965 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-30T21:41:43.965 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-30T21:41:43.965 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-30T21:41:43.965 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-30T21:41:43.965 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-30T21:41:43.965 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-30T21:41:43.965 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-30T21:41:43.965 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-30T21:41:44.079 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:41:44.323 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-30T21:41:44.620 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-30T21:41:44.957 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-30T21:41:44.958 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:44.958 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:41:44.958 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-30T21:41:44.958 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-30T21:41:44.958 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-30T21:41:44.958 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-30T21:41:44.958 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-30T21:41:44.958 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-30T21:41:44.958 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-30T21:41:44.958 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-30T21:41:44.959 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-30T21:41:44.959 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-30T21:41:44.959 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-30T21:41:44.959 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-30T21:41:44.959 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-30T21:41:44.959 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:44.959 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:41:45.086 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-cephadm 2023-11-30T21:41:45.578 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-30T21:41:45.957 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-30T21:41:45.980 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:14 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:41:46.157 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-30T21:41:46.367 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-30T21:41:46.675 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-30T21:41:46.675 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-30T21:41:46.675 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-30T21:41:46.675 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-30T21:41:46.675 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-30T21:41:46.676 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-30T21:41:47.394 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-30T21:41:47.394 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:47.395 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:41:47.395 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-30T21:41:47.395 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-30T21:41:47.395 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-30T21:41:47.395 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:47.395 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:41:47.548 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-fuse 2023-11-30T21:41:47.800 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:41:47.801 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:47.802 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-11-30T21:41:47.802 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:47.802 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:41:47.802 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-30T21:41:47.802 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-11-30T21:41:47.802 INFO:teuthology.orchestra.run.smithi027.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-30T21:41:47.802 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-30T21:41:47.802 INFO:teuthology.orchestra.run.smithi027.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-30T21:41:47.802 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-30T21:41:47.802 INFO:teuthology.orchestra.run.smithi027.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-30T21:41:47.802 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:47.803 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:41:47.803 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:47.803 INFO:teuthology.orchestra.run.smithi027.stdout:Install 6 Packages 2023-11-30T21:41:47.803 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:47.803 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 5.8 M 2023-11-30T21:41:47.803 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 24 M 2023-11-30T21:41:47.804 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:41:47.951 INFO:teuthology.orchestra.run.smithi027.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 89 kB 00:00 2023-11-30T21:41:48.026 INFO:teuthology.orchestra.run.smithi027.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 283 kB/s | 21 kB 00:00 2023-11-30T21:41:48.143 INFO:teuthology.orchestra.run.smithi027.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 3.0 MB/s | 355 kB 00:00 2023-11-30T21:41:48.185 INFO:teuthology.orchestra.run.smithi027.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 1.1 MB/s | 49 kB 00:00 2023-11-30T21:41:48.445 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:15 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:41:48.460 INFO:teuthology.orchestra.run.smithi027.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 966 kB/s | 538 kB 00:00 2023-11-30T21:41:49.239 INFO:teuthology.orchestra.run.smithi027.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 3.6 MB/s | 4.8 MB 00:01 2023-11-30T21:41:49.240 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:41:49.241 INFO:teuthology.orchestra.run.smithi027.stdout:Total 4.0 MB/s | 5.8 MB 00:01 2023-11-30T21:41:49.297 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:41:49.308 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:41:49.308 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:41:49.425 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:41:49.430 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:41:49.738 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:41:49.998 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-30T21:41:50.249 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:41:50.250 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:50.250 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-11-30T21:41:50.250 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:50.250 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:41:50.250 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-30T21:41:50.250 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2023-11-30T21:41:50.250 INFO:teuthology.orchestra.run.smithi049.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-30T21:41:50.250 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:50.250 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:41:50.250 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:50.250 INFO:teuthology.orchestra.run.smithi049.stdout:Install 2 Packages 2023-11-30T21:41:50.251 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:50.251 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 968 k 2023-11-30T21:41:50.251 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 3.2 M 2023-11-30T21:41:50.251 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:41:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2023-11-30T21:41:50.489 INFO:teuthology.orchestra.run.smithi049.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 422 kB/s | 83 kB 00:00 2023-11-30T21:41:50.490 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:41:50.490 INFO:teuthology.orchestra.run.smithi049.stdout:Total 4.0 MB/s | 968 kB 00:00 2023-11-30T21:41:50.496 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-30T21:41:50.500 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:41:50.522 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:41:50.522 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:41:50.613 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:41:50.617 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:41:50.792 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:41:51.064 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-30T21:41:51.233 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-30T21:41:51.488 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-30T21:41:51.639 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-30T21:41:51.640 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-30T21:41:51.937 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-30T21:41:52.207 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-30T21:41:52.417 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-30T21:41:52.426 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-30T21:41:52.426 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:52.426 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:41:52.426 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-30T21:41:52.426 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:52.426 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:41:52.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados-devel 2023-11-30T21:41:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-30T21:41:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-30T21:41:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-30T21:41:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-30T21:41:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-30T21:41:52.742 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-30T21:41:53.473 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-30T21:41:53.473 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:53.473 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:41:53.473 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-30T21:41:53.473 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-30T21:41:53.473 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-30T21:41:53.473 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:53.473 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:41:53.579 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:20 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:41:53.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-fuse 2023-11-30T21:41:54.500 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:23 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:41:55.388 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:41:55.389 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:55.389 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-11-30T21:41:55.390 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:55.390 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:41:55.390 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-30T21:41:55.390 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:55.390 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:41:55.390 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:41:55.391 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-30T21:41:55.391 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:55.391 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 131 k 2023-11-30T21:41:55.391 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 439 k 2023-11-30T21:41:55.391 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:41:55.436 INFO:teuthology.orchestra.run.smithi049.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.8 MB/s | 131 kB 00:00 2023-11-30T21:41:55.436 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:41:55.436 INFO:teuthology.orchestra.run.smithi049.stdout:Total 2.7 MB/s | 131 kB 00:00 2023-11-30T21:41:55.437 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:41:55.443 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:41:55.443 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:41:55.497 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:41:55.500 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:41:55.616 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:41:55.790 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:41:56.099 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:41:56.295 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:41:56.295 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:56.295 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2023-11-30T21:41:56.296 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:56.296 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:41:56.296 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-30T21:41:56.296 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-11-30T21:41:56.296 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-30T21:41:56.296 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:56.296 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:41:56.296 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:41:56.296 INFO:teuthology.orchestra.run.smithi027.stdout:Install 2 Packages 2023-11-30T21:41:56.296 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:56.297 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 968 k 2023-11-30T21:41:56.297 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 3.2 M 2023-11-30T21:41:56.297 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:41:56.564 INFO:teuthology.orchestra.run.smithi027.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 12 MB/s | 885 kB 00:00 2023-11-30T21:41:56.706 INFO:teuthology.orchestra.run.smithi027.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 385 kB/s | 83 kB 00:00 2023-11-30T21:41:56.706 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:41:56.706 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.3 MB/s | 968 kB 00:00 2023-11-30T21:41:56.718 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:41:56.740 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:41:56.740 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:41:56.813 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:41:56.813 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:56.813 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:41:56.813 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-30T21:41:56.813 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:41:56.813 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:41:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:41:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:41:57.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs2 2023-11-30T21:41:57.026 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:41:57.338 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-30T21:41:57.569 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-30T21:41:57.918 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:24 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:41:58.032 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-30T21:41:58.032 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-30T21:41:58.809 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-30T21:41:58.809 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:58.810 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:41:58.810 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-30T21:41:58.810 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:41:58.810 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:41:59.042 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados-devel 2023-11-30T21:41:59.478 INFO:teuthology.orchestra.run.smithi049.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:41:59.717 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:41:59.718 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-30T21:41:59.718 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:41:59.826 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs-devel 2023-11-30T21:41:59.946 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:28 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:42:00.726 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:27 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:42:01.773 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:42:01.773 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:01.774 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2023-11-30T21:42:01.774 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:01.774 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:42:01.774 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-30T21:42:01.774 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:01.774 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:42:01.774 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:01.774 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-11-30T21:42:01.774 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:01.775 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 131 k 2023-11-30T21:42:01.775 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 439 k 2023-11-30T21:42:01.775 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:42:01.838 INFO:teuthology.orchestra.run.smithi027.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 131 kB 00:00 2023-11-30T21:42:01.838 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:42:01.838 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.0 MB/s | 131 kB 00:00 2023-11-30T21:42:01.839 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:42:01.844 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:42:01.844 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:42:01.897 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:42:01.900 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:42:02.017 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:42:02.282 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:02.534 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:42:02.535 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:42:02.535 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-11-30T21:42:02.535 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:42:02.535 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:42:02.536 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-30T21:42:02.536 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:02.536 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:42:02.536 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:42:02.536 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-30T21:42:02.536 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:02.536 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 24 k 2023-11-30T21:42:02.536 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 94 k 2023-11-30T21:42:02.536 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:42:02.576 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:02.590 INFO:teuthology.orchestra.run.smithi049.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 452 kB/s | 24 kB 00:00 2023-11-30T21:42:02.590 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:42:02.591 INFO:teuthology.orchestra.run.smithi049.stdout:Total 442 kB/s | 24 kB 00:00 2023-11-30T21:42:02.591 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:42:02.594 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:42:02.594 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:42:02.597 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:42:02.600 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:42:02.662 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:42:02.839 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:03.118 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:03.277 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:03.277 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:03.277 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:42:03.277 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-30T21:42:03.278 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:03.278 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:42:03.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs2 2023-11-30T21:42:03.815 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:03.815 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:03.815 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:42:03.815 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-30T21:42:03.816 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:03.816 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:42:03.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados2 2023-11-30T21:42:04.440 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:33 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:42:04.836 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:31 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:42:05.989 INFO:teuthology.orchestra.run.smithi027.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:42:06.236 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:42:06.237 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-11-30T21:42:06.237 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:42:06.405 INFO:teuthology.orchestra.run.smithi049.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:42:06.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs-devel 2023-11-30T21:42:06.642 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:42:06.643 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-30T21:42:06.643 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:42:06.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librbd1 2023-11-30T21:42:07.452 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:36 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:42:07.650 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:34 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:42:09.191 INFO:teuthology.orchestra.run.smithi049.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:42:09.245 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:42:09.246 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:09.246 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2023-11-30T21:42:09.246 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:09.246 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:42:09.246 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-30T21:42:09.246 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:09.246 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:42:09.246 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:09.246 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-11-30T21:42:09.246 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:09.247 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 24 k 2023-11-30T21:42:09.247 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 94 k 2023-11-30T21:42:09.247 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:42:09.292 INFO:teuthology.orchestra.run.smithi027.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 542 kB/s | 24 kB 00:00 2023-11-30T21:42:09.292 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:42:09.292 INFO:teuthology.orchestra.run.smithi027.stdout:Total 528 kB/s | 24 kB 00:00 2023-11-30T21:42:09.293 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:42:09.296 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:42:09.297 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:42:09.299 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:42:09.302 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:42:09.355 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:42:09.431 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:42:09.433 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-30T21:42:09.433 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:42:09.571 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rados 2023-11-30T21:42:09.589 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:09.868 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:10.465 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:37 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:42:10.600 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:10.601 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:10.601 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:42:10.601 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-30T21:42:10.601 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:10.601 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:42:10.748 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados2 2023-11-30T21:42:11.649 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:40 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:42:12.039 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:42:12.302 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:42:12.303 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-30T21:42:12.304 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:42:12.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rgw 2023-11-30T21:42:13.218 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:42:13.341 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:40 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:42:13.460 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:42:13.462 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-11-30T21:42:13.462 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:42:13.582 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librbd1 2023-11-30T21:42:14.472 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:43 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:42:14.898 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:42:15.136 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:42:15.137 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-30T21:42:15.138 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:42:15.261 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-cephfs 2023-11-30T21:42:16.039 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:42:16.154 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:43 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:42:16.288 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:42:16.289 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-11-30T21:42:16.289 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:42:16.415 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rados 2023-11-30T21:42:17.312 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:46 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:42:17.741 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:42:17.981 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:42:17.982 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-30T21:42:17.983 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:42:18.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rbd 2023-11-30T21:42:18.885 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:42:19.005 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:46 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:42:19.123 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:42:19.125 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-11-30T21:42:19.126 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:42:19.239 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rgw 2023-11-30T21:42:20.139 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:49 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:42:20.558 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:42:20.797 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:42:20.798 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2023-11-30T21:42:20.798 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:42:20.915 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-fuse 2023-11-30T21:42:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:42:21.803 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:48 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:42:21.930 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:42:21.931 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-11-30T21:42:21.932 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:42:22.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-cephfs 2023-11-30T21:42:23.122 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:52 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:42:23.604 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:42:23.604 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:42:23.605 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-11-30T21:42:23.605 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:42:23.605 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:42:23.605 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-30T21:42:23.605 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:23.606 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:42:23.606 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:42:23.606 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-30T21:42:23.606 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:23.609 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 89 k 2023-11-30T21:42:23.610 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 298 k 2023-11-30T21:42:23.610 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:42:23.746 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 634 kB/s | 89 kB 00:00 2023-11-30T21:42:23.747 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:42:23.747 INFO:teuthology.orchestra.run.smithi049.stdout:Total 628 kB/s | 89 kB 00:00 2023-11-30T21:42:23.747 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:42:23.758 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:42:23.758 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:42:23.811 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:42:23.815 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:42:24.072 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:42:24.230 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:24.526 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:24.690 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:42:24.932 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:42:24.933 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-11-30T21:42:24.933 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:42:25.057 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rbd 2023-11-30T21:42:25.264 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:25.264 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:25.264 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:42:25.264 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-30T21:42:25.264 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:25.264 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:42:25.496 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-mirror 2023-11-30T21:42:25.951 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:54 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:42:26.395 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:53 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:42:27.511 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-30T21:42:27.758 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:42:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-11-30T21:42:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:42:28.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-fuse 2023-11-30T21:42:28.171 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:42:28.171 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:42:28.172 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-11-30T21:42:28.172 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:42:28.172 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:42:28.172 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-30T21:42:28.172 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:28.172 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:42:28.172 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:42:28.172 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-30T21:42:28.172 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:28.173 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 3.7 M 2023-11-30T21:42:28.173 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 18 M 2023-11-30T21:42:28.173 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:42:28.257 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 44 MB/s | 3.7 MB 00:00 2023-11-30T21:42:28.257 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:42:28.258 INFO:teuthology.orchestra.run.smithi049.stdout:Total 43 MB/s | 3.7 MB 00:00 2023-11-30T21:42:28.258 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:42:28.271 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:42:28.271 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:42:28.339 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:42:28.343 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:42:28.831 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:42:28.894 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:57 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:42:29.010 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:29.452 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:30.214 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:30.215 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:30.215 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:42:30.215 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-30T21:42:30.215 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:30.215 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:42:30.472 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-nbd 2023-11-30T21:42:30.686 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:42:30.688 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:30.689 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2023-11-30T21:42:30.689 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:30.689 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:42:30.689 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-30T21:42:30.689 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:30.689 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:42:30.689 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:30.689 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-11-30T21:42:30.689 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:30.689 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 89 k 2023-11-30T21:42:30.689 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 298 k 2023-11-30T21:42:30.690 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:42:30.741 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.7 MB/s | 89 kB 00:00 2023-11-30T21:42:30.741 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:42:30.742 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.6 MB/s | 89 kB 00:00 2023-11-30T21:42:30.742 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:42:30.752 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:42:30.753 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:42:30.805 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:42:30.808 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:42:30.928 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:42:31.153 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:31.357 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:58 ago on Thu 30 Nov 2023 09:39:33 PM UTC. 2023-11-30T21:42:31.453 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:32.159 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:32.160 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:32.160 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:42:32.160 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-30T21:42:32.160 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:32.160 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:42:32.411 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-mirror 2023-11-30T21:42:33.138 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2023-11-30T21:42:33.138 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2023-11-30T21:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2023-11-30T21:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-30T21:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2023-11-30T21:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2023-11-30T21:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2023-11-30T21:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 173 k 2023-11-30T21:42:33.139 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 649 k 2023-11-30T21:42:33.140 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2023-11-30T21:42:33.219 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 173 kB 00:00 2023-11-30T21:42:33.219 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:42:33.220 INFO:teuthology.orchestra.run.smithi049.stdout:Total 2.1 MB/s | 173 kB 00:00 2023-11-30T21:42:33.220 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2023-11-30T21:42:33.231 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2023-11-30T21:42:33.232 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2023-11-30T21:42:33.286 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2023-11-30T21:42:33.289 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2023-11-30T21:42:33.311 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:03:02 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:42:33.407 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2023-11-30T21:42:33.602 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:33.898 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:34.610 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:34.610 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:34.610 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2023-11-30T21:42:34.610 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-30T21:42:34.610 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:42:34.610 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2023-11-30T21:42:34.812 DEBUG:teuthology.parallel:result is None 2023-11-30T21:42:35.119 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:42:35.120 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:35.120 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2023-11-30T21:42:35.120 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:35.120 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:42:35.120 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-30T21:42:35.120 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:35.120 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:42:35.120 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:35.121 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-11-30T21:42:35.121 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:35.121 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 3.7 M 2023-11-30T21:42:35.121 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 18 M 2023-11-30T21:42:35.121 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:42:35.244 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2023-11-30T21:42:35.245 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:42:35.245 INFO:teuthology.orchestra.run.smithi027.stdout:Total 30 MB/s | 3.7 MB 00:00 2023-11-30T21:42:35.245 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:42:35.258 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:42:35.258 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:42:35.328 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:42:35.331 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:42:35.834 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:42:36.053 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:36.496 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:37.276 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:42:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-30T21:42:37.276 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:37.277 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:42:37.500 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-nbd 2023-11-30T21:42:38.393 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:03:07 ago on Thu 30 Nov 2023 09:39:31 PM UTC. 2023-11-30T21:42:40.192 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-11-30T21:42:40.193 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:40.193 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2023-11-30T21:42:40.193 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:40.193 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-11-30T21:42:40.193 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-30T21:42:40.193 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:40.193 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-11-30T21:42:40.193 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2023-11-30T21:42:40.194 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2023-11-30T21:42:40.194 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:40.194 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 173 k 2023-11-30T21:42:40.194 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 649 k 2023-11-30T21:42:40.194 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-11-30T21:42:40.239 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.8 MB/s | 173 kB 00:00 2023-11-30T21:42:40.239 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-11-30T21:42:40.240 INFO:teuthology.orchestra.run.smithi027.stdout:Total 3.7 MB/s | 173 kB 00:00 2023-11-30T21:42:40.240 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-11-30T21:42:40.251 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-11-30T21:42:40.251 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-11-30T21:42:40.306 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-11-30T21:42:40.309 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-11-30T21:42:40.425 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-11-30T21:42:40.673 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:40.952 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:41.654 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-30T21:42:41.654 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:41.654 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-11-30T21:42:41.654 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-30T21:42:41.654 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-11-30T21:42:41.654 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-11-30T21:42:41.913 DEBUG:teuthology.parallel:result is None 2023-11-30T21:42:41.914 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-30T21:42:41.914 INFO:teuthology.packaging:ref: None 2023-11-30T21:42:41.914 INFO:teuthology.packaging:tag: v16.2.4 2023-11-30T21:42:41.914 INFO:teuthology.packaging:branch: None 2023-11-30T21:42:41.914 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-11-30T21:42:41.915 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-30T21:42:42.143 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-30T21:42:42.166 INFO:teuthology.orchestra.run.smithi027.stdout:16.2.4-0.el8 2023-11-30T21:42:42.167 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-30T21:42:42.167 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-30T21:42:42.167 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-30T21:42:42.167 INFO:teuthology.packaging:ref: None 2023-11-30T21:42:42.167 INFO:teuthology.packaging:tag: v16.2.4 2023-11-30T21:42:42.167 INFO:teuthology.packaging:branch: None 2023-11-30T21:42:42.168 INFO:teuthology.packaging:sha1: 0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9 2023-11-30T21:42:42.168 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-30T21:42:42.307 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-30T21:42:42.336 INFO:teuthology.orchestra.run.smithi049.stdout:16.2.4-0.el8 2023-11-30T21:42:42.336 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-30T21:42:42.336 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-30T21:42:42.337 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-30T21:42:42.337 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-11-30T21:42:42.337 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-30T21:42:42.366 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-30T21:42:42.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-30T21:42:42.410 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-30T21:42:42.411 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-11-30T21:42:42.411 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2023-11-30T21:42:42.438 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-30T21:42:42.506 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-30T21:42:42.506 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/daemon-helper 2023-11-30T21:42:42.540 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-30T21:42:42.612 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-30T21:42:42.612 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-11-30T21:42:42.612 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-30T21:42:42.640 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-30T21:42:42.709 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-30T21:42:42.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-30T21:42:42.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-30T21:42:42.814 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-30T21:42:42.814 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-11-30T21:42:42.814 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2023-11-30T21:42:42.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-30T21:42:42.909 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-30T21:42:42.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/stdin-killer 2023-11-30T21:42:42.943 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-30T21:42:43.014 INFO:teuthology.run_tasks:Running task print... 2023-11-30T21:42:43.023 INFO:teuthology.task.print:**** done install task... 2023-11-30T21:42:43.023 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-30T21:42:43.157 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9'} 2023-11-30T21:42:43.157 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-11-30T21:42:43.157 INFO:tasks.cephadm:Cluster fsid is 6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:42:43.158 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-30T21:42:43.158 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-11-30T21:42:43.158 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi027': '172.21.15.27', 'mon.smithi049': '172.21.15.49'} 2023-11-30T21:42:43.158 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-30T21:42:43.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo hostname $(hostname -s) 2023-11-30T21:42:43.187 DEBUG:teuthology.orchestra.run.smithi049:> sudo hostname $(hostname -s) 2023-11-30T21:42:43.219 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-11-30T21:42:43.220 DEBUG:teuthology.orchestra.run.smithi027:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-30T21:42:43.506 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 30 21:42 /home/ubuntu/cephtest/cephadm 2023-11-30T21:42:43.506 DEBUG:teuthology.orchestra.run.smithi049:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-30T21:42:43.661 INFO:teuthology.orchestra.run.smithi049.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 30 21:42 /home/ubuntu/cephtest/cephadm 2023-11-30T21:42:43.662 DEBUG:teuthology.orchestra.run.smithi027:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-30T21:42:43.683 DEBUG:teuthology.orchestra.run.smithi049:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-30T21:42:43.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph 2023-11-30T21:42:43.752 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph 2023-11-30T21:42:43.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /etc/ceph 2023-11-30T21:42:43.820 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 777 /etc/ceph 2023-11-30T21:42:43.858 INFO:tasks.cephadm:Writing seed config... 2023-11-30T21:42:43.859 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-30T21:42:43.859 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-30T21:42:43.859 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-30T21:42:43.860 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-30T21:42:43.860 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-30T21:42:43.860 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-11-30T21:42:43.860 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-30T21:42:43.860 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-11-30T21:42:43.860 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-30T21:42:43.860 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-30T21:42:43.860 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-30T21:42:43.860 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-30T21:42:43.860 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-30T21:42:43.860 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-30T21:42:43.860 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-30T21:42:43.861 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-30T21:42:43.861 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-30T21:42:43.861 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-30T21:42:43.861 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-30T21:42:43.861 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-30T21:42:43.861 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-30T21:42:43.861 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-30T21:42:43.861 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-11-30T21:42:43.861 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-11-30T21:42:43.861 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-11-30T21:42:43.861 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-30T21:42:43.861 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-30T21:42:43.861 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-11-30T21:42:43.862 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-11-30T21:42:43.862 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-30T21:42:43.862 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-30T21:42:43.862 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-11-30T21:42:43.862 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-30T21:42:43.862 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-30T21:42:43.862 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-30T21:42:43.862 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-30T21:42:43.862 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-30T21:42:43.862 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-30T21:42:43.863 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-11-30T21:42:43.863 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-30T21:42:43.882 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 6473e080-8fc9-11ee-95a2-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-30T21:42:43.883 DEBUG:teuthology.orchestra.run.smithi027:mon.smithi027> sudo journalctl -f -n 0 -u ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi027.service 2023-11-30T21:42:43.925 INFO:tasks.cephadm:Bootstrapping... 2023-11-30T21:42:43.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.27 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-30T21:42:43.952 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:-- Logs begin at Thu 2023-11-30 21:19:13 UTC. -- 2023-11-30T21:42:44.205 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-30T21:42:45.310 INFO:teuthology.orchestra.run.smithi027.stderr:/bin/podman: 4.6.1 2023-11-30T21:42:45.321 INFO:teuthology.orchestra.run.smithi027.stderr:Verifying podman|docker is present... 2023-11-30T21:42:45.321 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-30T21:42:45.346 INFO:teuthology.orchestra.run.smithi027.stderr:/bin/podman: 4.6.1 2023-11-30T21:42:45.357 INFO:teuthology.orchestra.run.smithi027.stderr:Verifying lvm2 is present... 2023-11-30T21:42:45.357 INFO:teuthology.orchestra.run.smithi027.stderr:Verifying time synchronization is in place... 2023-11-30T21:42:45.357 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-enabled chrony.service 2023-11-30T21:42:45.364 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-30T21:42:45.365 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-active chrony.service 2023-11-30T21:42:45.371 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: inactive 2023-11-30T21:42:45.371 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-30T21:42:45.378 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: enabled 2023-11-30T21:42:45.378 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-active chronyd.service 2023-11-30T21:42:45.384 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: active 2023-11-30T21:42:45.385 INFO:teuthology.orchestra.run.smithi027.stderr:Unit chronyd.service is enabled and running 2023-11-30T21:42:45.385 INFO:teuthology.orchestra.run.smithi027.stderr:Repeating the final host check... 2023-11-30T21:42:45.385 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-30T21:42:45.406 INFO:teuthology.orchestra.run.smithi027.stderr:/bin/podman: 4.6.1 2023-11-30T21:42:45.414 INFO:teuthology.orchestra.run.smithi027.stderr:podman|docker (/bin/podman) is present 2023-11-30T21:42:45.414 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl is present 2023-11-30T21:42:45.414 INFO:teuthology.orchestra.run.smithi027.stderr:lvcreate is present 2023-11-30T21:42:45.414 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-enabled chrony.service 2023-11-30T21:42:45.420 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-30T21:42:45.420 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-active chrony.service 2023-11-30T21:42:45.426 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: inactive 2023-11-30T21:42:45.426 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-30T21:42:45.432 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: enabled 2023-11-30T21:42:45.433 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-active chronyd.service 2023-11-30T21:42:45.438 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: active 2023-11-30T21:42:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:Unit chronyd.service is enabled and running 2023-11-30T21:42:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:Host looks OK 2023-11-30T21:42:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:Cluster fsid: 6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:42:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:Acquiring lock 139937109555912 on /run/cephadm/6473e080-8fc9-11ee-95a2-87774f69a715.lock 2023-11-30T21:42:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:Lock 139937109555912 acquired on /run/cephadm/6473e080-8fc9-11ee-95a2-87774f69a715.lock 2023-11-30T21:42:45.440 INFO:teuthology.orchestra.run.smithi027.stderr:Verifying IP 172.21.15.27 port 3300 ... 2023-11-30T21:42:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:Verifying IP 172.21.15.27 port 6789 ... 2023-11-30T21:42:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:Base mon IP is 172.21.15.27, final addrv is [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2023-11-30T21:42:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /sbin/ip route ls 2023-11-30T21:42:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.27 metric 100 2023-11-30T21:42:45.443 INFO:teuthology.orchestra.run.smithi027.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.27 metric 100 2023-11-30T21:42:45.444 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /sbin/ip -6 route ls 2023-11-30T21:42:45.446 INFO:teuthology.orchestra.run.smithi027.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-11-30T21:42:45.446 INFO:teuthology.orchestra.run.smithi027.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-11-30T21:42:45.446 INFO:teuthology.orchestra.run.smithi027.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2023-11-30T21:42:45.446 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /sbin/ip -6 addr ls 2023-11-30T21:42:45.449 INFO:teuthology.orchestra.run.smithi027.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-30T21:42:45.449 INFO:teuthology.orchestra.run.smithi027.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-11-30T21:42:45.449 INFO:teuthology.orchestra.run.smithi027.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-30T21:42:45.449 INFO:teuthology.orchestra.run.smithi027.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2023-11-30T21:42:45.449 INFO:teuthology.orchestra.run.smithi027.stderr:/sbin/ip: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2023-11-30T21:42:45.449 INFO:teuthology.orchestra.run.smithi027.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-30T21:42:45.450 INFO:teuthology.orchestra.run.smithi027.stderr:Mon IP 172.21.15.27 is in CIDR network 172.21.0.0/20 2023-11-30T21:42:45.450 INFO:teuthology.orchestra.run.smithi027.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-30T21:42:45.451 INFO:teuthology.orchestra.run.smithi027.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-11-30T21:42:45.451 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-11-30T21:42:45.551 INFO:teuthology.orchestra.run.smithi027.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-11-30T21:42:52.056 INFO:teuthology.orchestra.run.smithi027.stderr:/bin/podman: Getting image source signatures 2023-11-30T21:42:52.062 INFO:teuthology.orchestra.run.smithi027.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-11-30T21:42:52.062 INFO:teuthology.orchestra.run.smithi027.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-11-30T21:43:04.445 INFO:teuthology.orchestra.run.smithi027.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-30T21:43:04.495 INFO:teuthology.orchestra.run.smithi027.stderr:/bin/podman: Writing manifest to image destination 2023-11-30T21:43:04.844 INFO:teuthology.orchestra.run.smithi027.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-30T21:43:04.859 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-11-30T21:43:05.439 INFO:teuthology.orchestra.run.smithi027.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-30T21:43:05.766 INFO:teuthology.orchestra.run.smithi027.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-30T21:43:05.766 INFO:teuthology.orchestra.run.smithi027.stderr:Extracting ceph user uid/gid from container image... 2023-11-30T21:43:05.766 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-11-30T21:43:06.037 INFO:teuthology.orchestra.run.smithi027.stderr:stat: 167 167 2023-11-30T21:43:06.359 INFO:teuthology.orchestra.run.smithi027.stderr:Creating initial keys... 2023-11-30T21:43:06.359 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-30T21:43:06.685 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-authtool: AQDqAWllV4ShJhAAF5y+45TVC/3UaPb711eFsQ== 2023-11-30T21:43:07.014 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-30T21:43:07.391 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-authtool: AQDrAWlll4J7FhAAETv/rAztMMxYd+Vztu6Edg== 2023-11-30T21:43:07.717 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-30T21:43:08.040 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-authtool: AQDsAWlljlFhABAAA+v2XuhZ7ax6YW+40KWrMg== 2023-11-30T21:43:08.369 INFO:teuthology.orchestra.run.smithi027.stderr:Creating initial monmap... 2023-11-30T21:43:08.370 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpduzlh_fs:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 --addv smithi027 [v2:172.21.15.27:3300,v1:172.21.15.27:6789] /tmp/monmap 2023-11-30T21:43:08.711 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-30T21:43:08.711 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:43:08.711 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-30T21:43:09.034 INFO:teuthology.orchestra.run.smithi027.stderr:monmaptool for smithi027 [v2:172.21.15.27:3300,v1:172.21.15.27:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-30T21:43:09.035 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/monmaptool: set fsid to 6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:43:09.035 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-30T21:43:09.035 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-11-30T21:43:09.035 INFO:teuthology.orchestra.run.smithi027.stderr:Creating mon... 2023-11-30T21:43:09.035 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/mon.smithi027:/var/lib/ceph/mon/ceph-smithi027:z -v /tmp/ceph-tmpgoqcv_ay:/tmp/keyring:z -v /tmp/ceph-tmpduzlh_fs:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi027 --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-11-30T21:43:09.381 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.368+0000 7f03ef42f700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-30T21:43:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 1 imported monmap: 2023-11-30T21:43:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: epoch 0 2023-11-30T21:43:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: fsid 6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:43:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: last_changed 2023-11-30T21:43:08.685486+0000 2023-11-30T21:43:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: created 2023-11-30T21:43:08.685486+0000 2023-11-30T21:43:09.408 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-11-30T21:43:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-11-30T21:43:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.smithi027 2023-11-30T21:43:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 0 /usr/bin/ceph-mon: set fsid to 6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:43:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: RocksDB version: 6.8.1 2023-11-30T21:43:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-30T21:43:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Compile date May 13 2021 2023-11-30T21:43:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: DB SUMMARY 2023-11-30T21:43:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi027/store.db dir, Total Num: 0, files: 2023-11-30T21:43:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.409 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi027/store.db: 2023-11-30T21:43:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.error_if_exists: 0 2023-11-30T21:43:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.create_if_missing: 1 2023-11-30T21:43:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.paranoid_checks: 1 2023-11-30T21:43:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.env: 0x5597d15b81c0 2023-11-30T21:43:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.fs: Posix File System 2023-11-30T21:43:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.info_log: 0x5597d21ceec0 2023-11-30T21:43:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-30T21:43:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.statistics: (nil) 2023-11-30T21:43:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.use_fsync: 0 2023-11-30T21:43:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.max_log_file_size: 0 2023-11-30T21:43:09.410 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-30T21:43:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-30T21:43:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-30T21:43:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-30T21:43:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.allow_fallocate: 1 2023-11-30T21:43:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-30T21:43:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-30T21:43:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.use_direct_reads: 0 2023-11-30T21:43:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-30T21:43:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-30T21:43:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.db_log_dir: 2023-11-30T21:43:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi027/store.db 2023-11-30T21:43:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-30T21:43:09.411 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.max_subcompactions: 1 2023-11-30T21:43:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.max_background_flushes: -1 2023-11-30T21:43:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-30T21:43:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-30T21:43:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-30T21:43:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-30T21:43:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-30T21:43:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-30T21:43:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-30T21:43:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.write_buffer_manager: 0x5597d21d5530 2023-11-30T21:43:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-30T21:43:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-30T21:43:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-30T21:43:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-30T21:43:09.412 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-30T21:43:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-30T21:43:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-30T21:43:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-30T21:43:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-30T21:43:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.unordered_write: 0 2023-11-30T21:43:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-30T21:43:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-30T21:43:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-30T21:43:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-30T21:43:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.row_cache: None 2023-11-30T21:43:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.wal_filter: None 2023-11-30T21:43:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-30T21:43:09.413 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-30T21:43:09.414 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.preserve_deletes: 0 2023-11-30T21:43:09.414 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.two_write_queues: 0 2023-11-30T21:43:09.414 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-30T21:43:09.414 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.atomic_flush: 0 2023-11-30T21:43:09.414 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-30T21:43:09.414 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-30T21:43:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-30T21:43:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.log_readahead_size: 0 2023-11-30T21:43:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-30T21:43:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.max_background_jobs: 2 2023-11-30T21:43:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.max_background_compactions: -1 2023-11-30T21:43:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-30T21:43:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-30T21:43:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-30T21:43:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-30T21:43:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-30T21:43:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-30T21:43:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-30T21:43:09.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-30T21:43:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.max_open_files: -1 2023-11-30T21:43:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-30T21:43:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-30T21:43:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-30T21:43:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-30T21:43:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Compression algorithms supported: 2023-11-30T21:43:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-30T21:43:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: kZSTD supported: 0 2023-11-30T21:43:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: kXpressCompression supported: 0 2023-11-30T21:43:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-30T21:43:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: kLZ4Compression supported: 1 2023-11-30T21:43:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: kBZip2Compression supported: 0 2023-11-30T21:43:09.416 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: kZlibCompression supported: 1 2023-11-30T21:43:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: kSnappyCompression supported: 1 2023-11-30T21:43:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.369+0000 7f03ef42f700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-30T21:43:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-11-30T21:43:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi027/store.db/MANIFEST-000001 2023-11-30T21:43:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-30T21:43:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-30T21:43:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.merge_operator: 2023-11-30T21:43:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compaction_filter: None 2023-11-30T21:43:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-30T21:43:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-30T21:43:09.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-30T21:43:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5597d20e6120) 2023-11-30T21:43:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-11-30T21:43:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-30T21:43:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-30T21:43:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-11-30T21:43:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: index_type: 0 2023-11-30T21:43:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-11-30T21:43:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-11-30T21:43:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-11-30T21:43:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-11-30T21:43:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: checksum: 1 2023-11-30T21:43:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-11-30T21:43:09.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: block_cache: 0x5597d211cd10 2023-11-30T21:43:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-11-30T21:43:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-11-30T21:43:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-11-30T21:43:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-11-30T21:43:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-11-30T21:43:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-11-30T21:43:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-11-30T21:43:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-11-30T21:43:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-11-30T21:43:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-11-30T21:43:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-11-30T21:43:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-11-30T21:43:09.419 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-11-30T21:43:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-11-30T21:43:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-11-30T21:43:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-30T21:43:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-11-30T21:43:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-11-30T21:43:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-11-30T21:43:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: format_version: 2 2023-11-30T21:43:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-11-30T21:43:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: block_align: 0 2023-11-30T21:43:09.420 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-30T21:43:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-30T21:43:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compression: NoCompression 2023-11-30T21:43:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-30T21:43:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-30T21:43:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-30T21:43:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.num_levels: 7 2023-11-30T21:43:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-30T21:43:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-30T21:43:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-30T21:43:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-30T21:43:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-30T21:43:09.421 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-30T21:43:09.422 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-30T21:43:09.422 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-30T21:43:09.422 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-30T21:43:09.422 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-30T21:43:09.422 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-30T21:43:09.422 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-30T21:43:09.422 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-30T21:43:09.422 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-30T21:43:09.422 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-30T21:43:09.422 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-30T21:43:09.423 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-30T21:43:09.423 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-30T21:43:09.423 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-30T21:43:09.423 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-30T21:43:09.423 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-30T21:43:09.423 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-30T21:43:09.423 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-30T21:43:09.423 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-30T21:43:09.423 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-30T21:43:09.423 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-30T21:43:09.423 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-30T21:43:09.423 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-30T21:43:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-30T21:43:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-30T21:43:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-30T21:43:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-30T21:43:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-30T21:43:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-30T21:43:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-30T21:43:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-30T21:43:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-30T21:43:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-30T21:43:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-30T21:43:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-30T21:43:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-30T21:43:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-30T21:43:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-30T21:43:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-30T21:43:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-30T21:43:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-30T21:43:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-30T21:43:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.table_properties_collectors: 2023-11-30T21:43:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.inplace_update_support: 0 2023-11-30T21:43:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-30T21:43:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-30T21:43:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-30T21:43:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-30T21:43:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.bloom_locality: 0 2023-11-30T21:43:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.max_successive_merges: 0 2023-11-30T21:43:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-30T21:43:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-30T21:43:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-30T21:43:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-30T21:43:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.ttl: 2592000 2023-11-30T21:43:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.370+0000 7f03ef42f700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-30T21:43:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.371+0000 7f03ef42f700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi027/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-30T21:43:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.371+0000 7f03ef42f700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-30T21:43:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.371+0000 7f03ef42f700 4 rocksdb: DB pointer 0x5597d21e1800 2023-11-30T21:43:09.426 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.371+0000 7f03d82c8700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-30T21:43:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.371+0000 7f03d82c8700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-30T21:43:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-11-30T21:43:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-30T21:43:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-30T21:43:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-30T21:43:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-30T21:43:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-30T21:43:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-30T21:43:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-30T21:43:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-30T21:43:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-30T21:43:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-30T21:43:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-30T21:43:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-30T21:43:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-30T21:43:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-30T21:43:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-30T21:43:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-30T21:43:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-30T21:43:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-30T21:43:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-30T21:43:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-30T21:43:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-30T21:43:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-30T21:43:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-30T21:43:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-30T21:43:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-30T21:43:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-30T21:43:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-30T21:43:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-30T21:43:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-30T21:43:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-30T21:43:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-30T21:43:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-30T21:43:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-30T21:43:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-30T21:43:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-30T21:43:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-30T21:43:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-30T21:43:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-30T21:43:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-30T21:43:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-30T21:43:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-30T21:43:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-30T21:43:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-30T21:43:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: 2023-11-30T21:43:09.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.373+0000 7f03ef42f700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-11-30T21:43:09.431 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.373+0000 7f03ef42f700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-11-30T21:43:09.431 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph-mon: debug 2023-11-30T21:43:09.373+0000 7f03ef42f700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi027 for mon.smithi027 2023-11-30T21:43:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:create mon.smithi027 on 2023-11-30T21:43:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:43:09.722 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl disable ceph.target 2023-11-30T21:43:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-30T21:43:09.820 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl enable ceph.target 2023-11-30T21:43:09.827 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-30T21:43:09.915 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl start ceph.target 2023-11-30T21:43:09.922 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl enable ceph-6473e080-8fc9-11ee-95a2-87774f69a715.target 2023-11-30T21:43:09.928 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-6473e080-8fc9-11ee-95a2-87774f69a715.target → /etc/systemd/system/ceph-6473e080-8fc9-11ee-95a2-87774f69a715.target. 2023-11-30T21:43:09.928 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-6473e080-8fc9-11ee-95a2-87774f69a715.target → /etc/systemd/system/ceph-6473e080-8fc9-11ee-95a2-87774f69a715.target. 2023-11-30T21:43:10.012 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl start ceph-6473e080-8fc9-11ee-95a2-87774f69a715.target 2023-11-30T21:43:10.020 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl daemon-reload 2023-11-30T21:43:10.111 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl stop ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi027 2023-11-30T21:43:10.138 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl reset-failed ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi027 2023-11-30T21:43:10.144 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: Failed to reset failed state of unit ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi027.service: Unit ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi027.service not loaded. 2023-11-30T21:43:10.144 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl enable ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi027 2023-11-30T21:43:10.151 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: Created symlink /etc/systemd/system/ceph-6473e080-8fc9-11ee-95a2-87774f69a715.target.wants/ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi027.service → /etc/systemd/system/ceph-6473e080-8fc9-11ee-95a2-87774f69a715@.service. 2023-11-30T21:43:10.243 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl start ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi027 2023-11-30T21:43:10.285 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:10 smithi027 systemd[1]: Starting Ceph mon.smithi027 for 6473e080-8fc9-11ee-95a2-87774f69a715... 2023-11-30T21:43:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-30T21:43:10.561 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: disabled 2023-11-30T21:43:10.561 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-active firewalld.service 2023-11-30T21:43:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: inactive 2023-11-30T21:43:10.569 INFO:teuthology.orchestra.run.smithi027.stderr:firewalld.service is not enabled 2023-11-30T21:43:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:Not possible to enable service . firewalld.service is not available 2023-11-30T21:43:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-30T21:43:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:Waiting for mon to start... 2023-11-30T21:43:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:Waiting for mon... 2023-11-30T21:43:10.570 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/mon.smithi027:/var/lib/ceph/mon/ceph-smithi027:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-11-30T21:43:10.815 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:10 smithi027 bash[140192]: 00c8b40cbcc2827489a4aafe0207bc62e42327b2b3b25496f3a791997017138e 2023-11-30T21:43:10.815 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:10 smithi027 systemd[1]: Started Ceph mon.smithi027 for 6473e080-8fc9-11ee-95a2-87774f69a715. 2023-11-30T21:43:10.815 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:10 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140206]: cluster 2023-11-30T21:43:10.587671+0000 mon.smithi027 (mon.0) 0 : [INF] mkfs 6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:43:10.815 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:10 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140206]: cluster 2023-11-30T21:43:10.567686+0000 mon.smithi027 (mon.0) 1 : cluster [INF] mon.smithi027 is new leader, mons smithi027 in quorum (ranks 0) 2023-11-30T21:43:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.037+0000 7f6ce92b8700 1 Processor -- start 2023-11-30T21:43:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.037+0000 7f6ce92b8700 1 -- start start 2023-11-30T21:43:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.037+0000 7f6ce92b8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce40fab90 0x7f6ce40fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.037+0000 7f6ce92b8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce40fb4a0 con 0x7f6ce40fab90 2023-11-30T21:43:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.038+0000 7f6ce2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce40fab90 0x7f6ce40fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.038+0000 7f6ce2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce40fab90 0x7f6ce40fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56624/0 (socket says 172.21.15.27:56624) 2023-11-30T21:43:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.038+0000 7f6ce2ffd700 1 -- 172.21.15.27:0/3028986165 learned_addr learned my addr 172.21.15.27:0/3028986165 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.038+0000 7f6ce2ffd700 1 -- 172.21.15.27:0/3028986165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce40fb5e0 con 0x7f6ce40fab90 2023-11-30T21:43:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.038+0000 7f6ce2ffd700 1 --2- 172.21.15.27:0/3028986165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce40fab90 0x7f6ce40fafb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f6cd400b5f0 tx=0x7f6cd4006ee0).ready entity=mon.0 client_cookie=929ec2239f3eaab3 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.038+0000 7f6ce1ffb700 1 -- 172.21.15.27:0/3028986165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cd400d600 con 0x7f6ce40fab90 2023-11-30T21:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.039+0000 7f6ce1ffb700 1 -- 172.21.15.27:0/3028986165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd400d760 con 0x7f6ce40fab90 2023-11-30T21:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.039+0000 7f6ce1ffb700 1 -- 172.21.15.27:0/3028986165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cd400da30 con 0x7f6ce40fab90 2023-11-30T21:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.039+0000 7f6ce92b8700 1 -- 172.21.15.27:0/3028986165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce40fab90 msgr2=0x7f6ce40fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.039+0000 7f6ce92b8700 1 --2- 172.21.15.27:0/3028986165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce40fab90 0x7f6ce40fafb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f6cd400b5f0 tx=0x7f6cd4006ee0).stop 2023-11-30T21:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.039+0000 7f6ce92b8700 1 -- 172.21.15.27:0/3028986165 shutdown_connections 2023-11-30T21:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.039+0000 7f6ce92b8700 1 --2- 172.21.15.27:0/3028986165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce40fab90 0x7f6ce40fafb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.039+0000 7f6ce92b8700 1 -- 172.21.15.27:0/3028986165 >> 172.21.15.27:0/3028986165 conn(0x7f6ce40f63b0 msgr2=0x7f6ce40f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.039+0000 7f6ce92b8700 1 -- 172.21.15.27:0/3028986165 shutdown_connections 2023-11-30T21:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.039+0000 7f6ce92b8700 1 -- 172.21.15.27:0/3028986165 wait complete. 2023-11-30T21:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.040+0000 7f6ce92b8700 1 Processor -- start 2023-11-30T21:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.040+0000 7f6ce92b8700 1 -- start start 2023-11-30T21:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.040+0000 7f6ce92b8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce40fab90 0x7f6ce410cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.040+0000 7f6ce92b8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce4070520 con 0x7f6ce40fab90 2023-11-30T21:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.040+0000 7f6ce2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce40fab90 0x7f6ce410cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.040+0000 7f6ce2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce40fab90 0x7f6ce410cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56634/0 (socket says 172.21.15.27:56634) 2023-11-30T21:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.040+0000 7f6ce2ffd700 1 -- 172.21.15.27:0/2328474656 learned_addr learned my addr 172.21.15.27:0/2328474656 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.041+0000 7f6ce2ffd700 1 -- 172.21.15.27:0/2328474656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd400b200 con 0x7f6ce40fab90 2023-11-30T21:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.041+0000 7f6ce2ffd700 1 --2- 172.21.15.27:0/2328474656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce40fab90 0x7f6ce410cf20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f6cd4006c70 tx=0x7f6cd4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.041+0000 7f6cdb7fe700 1 -- 172.21.15.27:0/2328474656 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cd4011b40 con 0x7f6ce40fab90 2023-11-30T21:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.041+0000 7f6cdb7fe700 1 -- 172.21.15.27:0/2328474656 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd4011ca0 con 0x7f6ce40fab90 2023-11-30T21:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.041+0000 7f6ce92b8700 1 -- 172.21.15.27:0/2328474656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce4070720 con 0x7f6ce40fab90 2023-11-30T21:43:11.045 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.041+0000 7f6cdb7fe700 1 -- 172.21.15.27:0/2328474656 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cd401d430 con 0x7f6ce40fab90 2023-11-30T21:43:11.045 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.041+0000 7f6ce92b8700 1 -- 172.21.15.27:0/2328474656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce410d5d0 con 0x7f6ce40fab90 2023-11-30T21:43:11.045 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.041+0000 7f6cdb7fe700 1 -- 172.21.15.27:0/2328474656 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f6cd400f070 con 0x7f6ce40fab90 2023-11-30T21:43:11.045 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.041+0000 7f6cdb7fe700 1 -- 172.21.15.27:0/2328474656 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6cd4027460 con 0x7f6ce40fab90 2023-11-30T21:43:11.045 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.042+0000 7f6ce92b8700 1 -- 172.21.15.27:0/2328474656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce4045ad0 con 0x7f6ce40fab90 2023-11-30T21:43:11.045 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.044+0000 7f6cdb7fe700 1 -- 172.21.15.27:0/2328474656 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f6cd4016070 con 0x7f6ce40fab90 2023-11-30T21:43:11.116 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.113+0000 7f6ce92b8700 1 -- 172.21.15.27:0/2328474656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f6ce40fbed0 con 0x7f6ce40fab90 2023-11-30T21:43:11.116 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.113+0000 7f6cdb7fe700 1 -- 172.21.15.27:0/2328474656 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f6cd4035160 con 0x7f6ce40fab90 2023-11-30T21:43:11.116 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: cluster: 2023-11-30T21:43:11.116 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: id: 6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:43:11.116 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-11-30T21:43:11.116 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: services: 2023-11-30T21:43:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi027 (age 0.528849s) 2023-11-30T21:43:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: mgr: no daemons active 2023-11-30T21:43:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-11-30T21:43:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: data: 2023-11-30T21:43:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-11-30T21:43:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-11-30T21:43:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-11-30T21:43:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: pgs: 2023-11-30T21:43:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.114+0000 7f6ce92b8700 1 -- 172.21.15.27:0/2328474656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce40fab90 msgr2=0x7f6ce410cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.114+0000 7f6ce92b8700 1 --2- 172.21.15.27:0/2328474656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce40fab90 0x7f6ce410cf20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f6cd4006c70 tx=0x7f6cd4016000).stop 2023-11-30T21:43:11.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.114+0000 7f6ce92b8700 1 -- 172.21.15.27:0/2328474656 shutdown_connections 2023-11-30T21:43:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.114+0000 7f6ce92b8700 1 --2- 172.21.15.27:0/2328474656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ce40fab90 0x7f6ce410cf20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.114+0000 7f6ce92b8700 1 -- 172.21.15.27:0/2328474656 >> 172.21.15.27:0/2328474656 conn(0x7f6ce40f63b0 msgr2=0x7f6ce40fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.114+0000 7f6ce92b8700 1 -- 172.21.15.27:0/2328474656 shutdown_connections 2023-11-30T21:43:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.114+0000 7f6ce92b8700 1 -- 172.21.15.27:0/2328474656 wait complete. 2023-11-30T21:43:11.467 INFO:teuthology.orchestra.run.smithi027.stderr:mon is available 2023-11-30T21:43:11.467 INFO:teuthology.orchestra.run.smithi027.stderr:Assimilating anything we can from ceph.conf... 2023-11-30T21:43:11.467 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z -v /var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/mon.smithi027:/var/lib/ceph/mon/ceph-smithi027:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi027/config 2023-11-30T21:43:11.680 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140206]: cluster 2023-11-30T21:43:10.586417+0000 mon.smithi027 (mon.0) 2 : cluster [INF] mon.smithi027 is new leader, mons smithi027 in quorum (ranks 0) 2023-11-30T21:43:11.680 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140206]: cluster 2023-11-30T21:43:10.586658+0000 mon.smithi027 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi027=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0]} 2023-11-30T21:43:11.680 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140206]: cluster 2023-11-30T21:43:10.589836+0000 mon.smithi027 (mon.0) 4 : cluster [DBG] fsmap 2023-11-30T21:43:11.680 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140206]: cluster 2023-11-30T21:43:10.592141+0000 mon.smithi027 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-30T21:43:11.680 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140206]: cluster 2023-11-30T21:43:10.592645+0000 mon.smithi027 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-11-30T21:43:11.680 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140206]: audit 2023-11-30T21:43:11.115223+0000 mon.smithi027 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.27:0/2328474656' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-30T21:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.959+0000 7f0869a6c700 1 Processor -- start 2023-11-30T21:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.959+0000 7f0869a6c700 1 -- start start 2023-11-30T21:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.960+0000 7f0869a6c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08640703f0 0x7f08640fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.960+0000 7f0869a6c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08640fee10 con 0x7f08640703f0 2023-11-30T21:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.960+0000 7f08637fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08640703f0 0x7f08640fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.960+0000 7f08637fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08640703f0 0x7f08640fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56640/0 (socket says 172.21.15.27:56640) 2023-11-30T21:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.960+0000 7f08637fe700 1 -- 172.21.15.27:0/1496313599 learned_addr learned my addr 172.21.15.27:0/1496313599 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.960+0000 7f08637fe700 1 -- 172.21.15.27:0/1496313599 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08640fef50 con 0x7f08640703f0 2023-11-30T21:43:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.960+0000 7f08637fe700 1 --2- 172.21.15.27:0/1496313599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08640703f0 0x7f08640fe920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f085400b5f0 tx=0x7f0854006ee0).ready entity=mon.0 client_cookie=dd040eaef6b33ec8 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.960+0000 7f08627fc700 1 -- 172.21.15.27:0/1496313599 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f085400d600 con 0x7f08640703f0 2023-11-30T21:43:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.960+0000 7f08627fc700 1 -- 172.21.15.27:0/1496313599 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085400d760 con 0x7f08640703f0 2023-11-30T21:43:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.960+0000 7f08627fc700 1 -- 172.21.15.27:0/1496313599 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f085400da30 con 0x7f08640703f0 2023-11-30T21:43:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.961+0000 7f0869a6c700 1 -- 172.21.15.27:0/1496313599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08640703f0 msgr2=0x7f08640fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.961+0000 7f0869a6c700 1 --2- 172.21.15.27:0/1496313599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08640703f0 0x7f08640fe920 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f085400b5f0 tx=0x7f0854006ee0).stop 2023-11-30T21:43:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.961+0000 7f0869a6c700 1 -- 172.21.15.27:0/1496313599 shutdown_connections 2023-11-30T21:43:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.961+0000 7f0869a6c700 1 --2- 172.21.15.27:0/1496313599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08640703f0 0x7f08640fe920 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.961+0000 7f0869a6c700 1 -- 172.21.15.27:0/1496313599 >> 172.21.15.27:0/1496313599 conn(0x7f08640f63c0 msgr2=0x7f08640f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.961+0000 7f0869a6c700 1 -- 172.21.15.27:0/1496313599 shutdown_connections 2023-11-30T21:43:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.961+0000 7f0869a6c700 1 -- 172.21.15.27:0/1496313599 wait complete. 2023-11-30T21:43:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.962+0000 7f0869a6c700 1 Processor -- start 2023-11-30T21:43:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.962+0000 7f0869a6c700 1 -- start start 2023-11-30T21:43:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.962+0000 7f0869a6c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08640703f0 0x7f086410a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.962+0000 7f0869a6c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086410a770 con 0x7f08640703f0 2023-11-30T21:43:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.962+0000 7f08637fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08640703f0 0x7f086410a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.962+0000 7f08637fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08640703f0 0x7f086410a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56648/0 (socket says 172.21.15.27:56648) 2023-11-30T21:43:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.962+0000 7f08637fe700 1 -- 172.21.15.27:0/1167313741 learned_addr learned my addr 172.21.15.27:0/1167313741 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.962+0000 7f08637fe700 1 -- 172.21.15.27:0/1167313741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f085400b200 con 0x7f08640703f0 2023-11-30T21:43:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.962+0000 7f08637fe700 1 --2- 172.21.15.27:0/1167313741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08640703f0 0x7f086410a280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f0854006c70 tx=0x7f0854016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.963+0000 7f085bfff700 1 -- 172.21.15.27:0/1167313741 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0854011b40 con 0x7f08640703f0 2023-11-30T21:43:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.963+0000 7f085bfff700 1 -- 172.21.15.27:0/1167313741 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0854011ca0 con 0x7f08640703f0 2023-11-30T21:43:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.963+0000 7f0869a6c700 1 -- 172.21.15.27:0/1167313741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086410a970 con 0x7f08640703f0 2023-11-30T21:43:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.963+0000 7f085bfff700 1 -- 172.21.15.27:0/1167313741 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f085401d430 con 0x7f08640703f0 2023-11-30T21:43:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.963+0000 7f0869a6c700 1 -- 172.21.15.27:0/1167313741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086410ad80 con 0x7f08640703f0 2023-11-30T21:43:11.969 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.963+0000 7f085bfff700 1 -- 172.21.15.27:0/1167313741 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f085400f070 con 0x7f08640703f0 2023-11-30T21:43:11.969 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.963+0000 7f085bfff700 1 -- 172.21.15.27:0/1167313741 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0854027460 con 0x7f08640703f0 2023-11-30T21:43:11.969 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.964+0000 7f0869a6c700 1 -- 172.21.15.27:0/1167313741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0864045ad0 con 0x7f08640703f0 2023-11-30T21:43:11.969 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:11.966+0000 7f085bfff700 1 -- 172.21.15.27:0/1167313741 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f0854016070 con 0x7f08640703f0 2023-11-30T21:43:12.036 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.034+0000 7f0869a6c700 1 -- 172.21.15.27:0/1167313741 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f08640ff1a0 con 0x7f08640703f0 2023-11-30T21:43:12.041 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.037+0000 7f085bfff700 1 -- 172.21.15.27:0/1167313741 <== mon.0 v2:172.21.15.27:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f085401d5e0 con 0x7f08640703f0 2023-11-30T21:43:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.037+0000 7f085bfff700 1 -- 172.21.15.27:0/1167313741 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f0854036020 con 0x7f08640703f0 2023-11-30T21:43:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: [global] 2023-11-30T21:43:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: fsid = 6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:43:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2023-11-30T21:43:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-30T21:43:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-30T21:43:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-30T21:43:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-30T21:43:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: [mgr] 2023-11-30T21:43:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-30T21:43:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: [osd] 2023-11-30T21:43:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-11-30T21:43:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-11-30T21:43:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-11-30T21:43:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-30T21:43:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-11-30T21:43:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-30T21:43:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.038+0000 7f0869a6c700 1 -- 172.21.15.27:0/1167313741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08640703f0 msgr2=0x7f086410a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:12.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.038+0000 7f0869a6c700 1 --2- 172.21.15.27:0/1167313741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08640703f0 0x7f086410a280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f0854006c70 tx=0x7f0854016000).stop 2023-11-30T21:43:12.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.038+0000 7f0869a6c700 1 -- 172.21.15.27:0/1167313741 shutdown_connections 2023-11-30T21:43:12.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.038+0000 7f0869a6c700 1 --2- 172.21.15.27:0/1167313741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08640703f0 0x7f086410a280 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:12.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.038+0000 7f0869a6c700 1 -- 172.21.15.27:0/1167313741 >> 172.21.15.27:0/1167313741 conn(0x7f08640f63c0 msgr2=0x7f08640f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:12.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.038+0000 7f0869a6c700 1 -- 172.21.15.27:0/1167313741 shutdown_connections 2023-11-30T21:43:12.044 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.038+0000 7f0869a6c700 1 -- 172.21.15.27:0/1167313741 wait complete. 2023-11-30T21:43:12.384 INFO:teuthology.orchestra.run.smithi027.stderr:Generating new minimal ceph.conf... 2023-11-30T21:43:12.384 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z -v /var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/mon.smithi027:/var/lib/ceph/mon/ceph-smithi027:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi027/config 2023-11-30T21:43:12.731 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.727+0000 7fc99c681700 1 Processor -- start 2023-11-30T21:43:12.731 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.727+0000 7fc99c681700 1 -- start start 2023-11-30T21:43:12.731 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.727+0000 7fc99c681700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940fab80 0x7fc9940fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:12.731 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.727+0000 7fc99c681700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9940fb490 con 0x7fc9940fab80 2023-11-30T21:43:12.731 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.727+0000 7fc99a41d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940fab80 0x7fc9940fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:12.731 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.727+0000 7fc99a41d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940fab80 0x7fc9940fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45842/0 (socket says 172.21.15.27:45842) 2023-11-30T21:43:12.731 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.727+0000 7fc99a41d700 1 -- 172.21.15.27:0/1299944943 learned_addr learned my addr 172.21.15.27:0/1299944943 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.728+0000 7fc99a41d700 1 -- 172.21.15.27:0/1299944943 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9940fb5d0 con 0x7fc9940fab80 2023-11-30T21:43:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.728+0000 7fc99a41d700 1 --2- 172.21.15.27:0/1299944943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940fab80 0x7fc9940fafa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fc99000b5f0 tx=0x7fc990006ee0).ready entity=mon.0 client_cookie=cb7dd89538383b4a server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.728+0000 7fc99941b700 1 -- 172.21.15.27:0/1299944943 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc99000d600 con 0x7fc9940fab80 2023-11-30T21:43:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.728+0000 7fc99941b700 1 -- 172.21.15.27:0/1299944943 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc99000dbd0 con 0x7fc9940fab80 2023-11-30T21:43:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.728+0000 7fc99941b700 1 -- 172.21.15.27:0/1299944943 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc990011c00 con 0x7fc9940fab80 2023-11-30T21:43:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.728+0000 7fc99c681700 1 -- 172.21.15.27:0/1299944943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940fab80 msgr2=0x7fc9940fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.728+0000 7fc99c681700 1 --2- 172.21.15.27:0/1299944943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940fab80 0x7fc9940fafa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fc99000b5f0 tx=0x7fc990006ee0).stop 2023-11-30T21:43:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.728+0000 7fc99c681700 1 -- 172.21.15.27:0/1299944943 shutdown_connections 2023-11-30T21:43:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.728+0000 7fc99c681700 1 --2- 172.21.15.27:0/1299944943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940fab80 0x7fc9940fafa0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.728+0000 7fc99c681700 1 -- 172.21.15.27:0/1299944943 >> 172.21.15.27:0/1299944943 conn(0x7fc9940f63c0 msgr2=0x7fc9940f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.729+0000 7fc99c681700 1 -- 172.21.15.27:0/1299944943 shutdown_connections 2023-11-30T21:43:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.729+0000 7fc99c681700 1 -- 172.21.15.27:0/1299944943 wait complete. 2023-11-30T21:43:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.729+0000 7fc99c681700 1 Processor -- start 2023-11-30T21:43:12.733 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.729+0000 7fc99c681700 1 -- start start 2023-11-30T21:43:12.734 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.729+0000 7fc99c681700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940fab80 0x7fc99410d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:12.734 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.729+0000 7fc99c681700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9940704b0 con 0x7fc9940fab80 2023-11-30T21:43:12.734 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.729+0000 7fc99a41d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940fab80 0x7fc99410d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:12.734 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.729+0000 7fc99a41d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940fab80 0x7fc99410d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45846/0 (socket says 172.21.15.27:45846) 2023-11-30T21:43:12.734 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.729+0000 7fc99a41d700 1 -- 172.21.15.27:0/482116817 learned_addr learned my addr 172.21.15.27:0/482116817 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:12.735 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.730+0000 7fc99a41d700 1 -- 172.21.15.27:0/482116817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99000b200 con 0x7fc9940fab80 2023-11-30T21:43:12.735 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.730+0000 7fc99a41d700 1 --2- 172.21.15.27:0/482116817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940fab80 0x7fc99410d6e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc99000e040 tx=0x7fc990006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:12.735 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.730+0000 7fc982ffd700 1 -- 172.21.15.27:0/482116817 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc99001b410 con 0x7fc9940fab80 2023-11-30T21:43:12.735 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.730+0000 7fc982ffd700 1 -- 172.21.15.27:0/482116817 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc99001b9e0 con 0x7fc9940fab80 2023-11-30T21:43:12.735 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.730+0000 7fc99c681700 1 -- 172.21.15.27:0/482116817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9940706b0 con 0x7fc9940fab80 2023-11-30T21:43:12.735 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.730+0000 7fc982ffd700 1 -- 172.21.15.27:0/482116817 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc99001a820 con 0x7fc9940fab80 2023-11-30T21:43:12.735 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.730+0000 7fc99c681700 1 -- 172.21.15.27:0/482116817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99410dd90 con 0x7fc9940fab80 2023-11-30T21:43:12.735 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.730+0000 7fc982ffd700 1 -- 172.21.15.27:0/482116817 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fc99000f070 con 0x7fc9940fab80 2023-11-30T21:43:12.735 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.730+0000 7fc982ffd700 1 -- 172.21.15.27:0/482116817 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fc990025b30 con 0x7fc9940fab80 2023-11-30T21:43:12.735 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.732+0000 7fc99c681700 1 -- 172.21.15.27:0/482116817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc994045ad0 con 0x7fc9940fab80 2023-11-30T21:43:12.736 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.734+0000 7fc982ffd700 1 -- 172.21.15.27:0/482116817 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fc990016070 con 0x7fc9940fab80 2023-11-30T21:43:12.760 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:12 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140206]: audit 2023-11-30T21:43:12.036133+0000 mon.smithi027 (mon.0) 8 : audit [INF] from='client.? 172.21.15.27:0/1167313741' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-30T21:43:12.760 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:12 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140206]: audit 2023-11-30T21:43:12.039170+0000 mon.smithi027 (mon.0) 9 : audit [INF] from='client.? 172.21.15.27:0/1167313741' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-30T21:43:12.803 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.801+0000 7fc99c681700 1 -- 172.21.15.27:0/482116817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fc994027a00 con 0x7fc9940fab80 2023-11-30T21:43:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.801+0000 7fc982ffd700 1 -- 172.21.15.27:0/482116817 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fc99001ad70 con 0x7fc9940fab80 2023-11-30T21:43:13.034 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.802+0000 7fc99c681700 1 -- 172.21.15.27:0/482116817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940fab80 msgr2=0x7fc99410d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.802+0000 7fc99c681700 1 --2- 172.21.15.27:0/482116817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940fab80 0x7fc99410d6e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc99000e040 tx=0x7fc990006ee0).stop 2023-11-30T21:43:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.802+0000 7fc99c681700 1 -- 172.21.15.27:0/482116817 shutdown_connections 2023-11-30T21:43:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.802+0000 7fc99c681700 1 --2- 172.21.15.27:0/482116817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9940fab80 0x7fc99410d6e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.802+0000 7fc99c681700 1 -- 172.21.15.27:0/482116817 >> 172.21.15.27:0/482116817 conn(0x7fc9940f63c0 msgr2=0x7fc9940f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.802+0000 7fc99c681700 1 -- 172.21.15.27:0/482116817 shutdown_connections 2023-11-30T21:43:13.035 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:12.802+0000 7fc99c681700 1 -- 172.21.15.27:0/482116817 wait complete. 2023-11-30T21:43:13.142 INFO:teuthology.orchestra.run.smithi027.stderr:Restarting the monitor... 2023-11-30T21:43:13.142 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl restart ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi027 2023-11-30T21:43:13.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:13 smithi027 systemd[1]: Stopping Ceph mon.smithi027 for 6473e080-8fc9-11ee-95a2-87774f69a715... 2023-11-30T21:43:13.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:13 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140206]: debug 2023-11-30T21:43:13.234+0000 7fb8bd3e6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi027 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-30T21:43:13.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:13 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140206]: debug 2023-11-30T21:43:13.234+0000 7fb8bd3e6700 -1 mon.smithi027@0(leader) e1 *** Got Signal Terminated *** 2023-11-30T21:43:13.981 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:13 smithi027 podman[140531]: ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027 2023-11-30T21:43:13.981 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:13 smithi027 systemd[1]: ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi027.service: Succeeded. 2023-11-30T21:43:13.981 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:13 smithi027 systemd[1]: Stopped Ceph mon.smithi027 for 6473e080-8fc9-11ee-95a2-87774f69a715. 2023-11-30T21:43:13.981 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:13 smithi027 systemd[1]: Starting Ceph mon.smithi027 for 6473e080-8fc9-11ee-95a2-87774f69a715... 2023-11-30T21:43:14.026 INFO:teuthology.orchestra.run.smithi027.stderr:Setting mon public_network to 172.21.0.0/20 2023-11-30T21:43:14.027 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-11-30T21:43:14.471 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.468+0000 7fdc3079c700 1 Processor -- start 2023-11-30T21:43:14.471 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.468+0000 7fdc3079c700 1 -- start start 2023-11-30T21:43:14.471 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.469+0000 7fdc3079c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc280703b0 0x7fdc280fe960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:14.471 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.469+0000 7fdc3079c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc280fee50 con 0x7fdc280703b0 2023-11-30T21:43:14.471 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.469+0000 7fdc2e538700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc280703b0 0x7fdc280fe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:14.472 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.469+0000 7fdc2e538700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc280703b0 0x7fdc280fe960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45858/0 (socket says 172.21.15.27:45858) 2023-11-30T21:43:14.472 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.469+0000 7fdc2e538700 1 -- 172.21.15.27:0/3421554951 learned_addr learned my addr 172.21.15.27:0/3421554951 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:14.472 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.470+0000 7fdc2e538700 1 -- 172.21.15.27:0/3421554951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc280fef90 con 0x7fdc280703b0 2023-11-30T21:43:14.472 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.470+0000 7fdc2e538700 1 --2- 172.21.15.27:0/3421554951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc280703b0 0x7fdc280fe960 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fdc1800b5f0 tx=0x7fdc18006ee0).ready entity=mon.0 client_cookie=9a85cb71a1ca8f00 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:14.472 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.470+0000 7fdc2d536700 1 -- 172.21.15.27:0/3421554951 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc1800d600 con 0x7fdc280703b0 2023-11-30T21:43:14.472 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.470+0000 7fdc2d536700 1 -- 172.21.15.27:0/3421554951 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdc1800dbd0 con 0x7fdc280703b0 2023-11-30T21:43:14.472 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.470+0000 7fdc2d536700 1 -- 172.21.15.27:0/3421554951 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc18011c00 con 0x7fdc280703b0 2023-11-30T21:43:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.470+0000 7fdc3079c700 1 -- 172.21.15.27:0/3421554951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc280703b0 msgr2=0x7fdc280fe960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.470+0000 7fdc3079c700 1 --2- 172.21.15.27:0/3421554951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc280703b0 0x7fdc280fe960 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fdc1800b5f0 tx=0x7fdc18006ee0).stop 2023-11-30T21:43:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.471+0000 7fdc3079c700 1 -- 172.21.15.27:0/3421554951 shutdown_connections 2023-11-30T21:43:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.471+0000 7fdc3079c700 1 --2- 172.21.15.27:0/3421554951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc280703b0 0x7fdc280fe960 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.471+0000 7fdc3079c700 1 -- 172.21.15.27:0/3421554951 >> 172.21.15.27:0/3421554951 conn(0x7fdc280f63c0 msgr2=0x7fdc280f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.471+0000 7fdc3079c700 1 -- 172.21.15.27:0/3421554951 shutdown_connections 2023-11-30T21:43:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.471+0000 7fdc3079c700 1 -- 172.21.15.27:0/3421554951 wait complete. 2023-11-30T21:43:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.471+0000 7fdc3079c700 1 Processor -- start 2023-11-30T21:43:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.471+0000 7fdc3079c700 1 -- start start 2023-11-30T21:43:14.475 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.471+0000 7fdc3079c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc280703b0 0x7fdc280fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:14.476 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.471+0000 7fdc3079c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc280fa860 con 0x7fdc280703b0 2023-11-30T21:43:14.476 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.472+0000 7fdc2e538700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc280703b0 0x7fdc280fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:14.476 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.472+0000 7fdc2e538700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc280703b0 0x7fdc280fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45868/0 (socket says 172.21.15.27:45868) 2023-11-30T21:43:14.476 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.472+0000 7fdc2e538700 1 -- 172.21.15.27:0/3654150879 learned_addr learned my addr 172.21.15.27:0/3654150879 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:14.476 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.472+0000 7fdc2e538700 1 -- 172.21.15.27:0/3654150879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1800b200 con 0x7fdc280703b0 2023-11-30T21:43:14.476 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.472+0000 7fdc2e538700 1 --2- 172.21.15.27:0/3654150879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc280703b0 0x7fdc280fc3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fdc1800b190 tx=0x7fdc18006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:14.476 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.472+0000 7fdc16ffd700 1 -- 172.21.15.27:0/3654150879 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc1801b410 con 0x7fdc280703b0 2023-11-30T21:43:14.476 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.472+0000 7fdc16ffd700 1 -- 172.21.15.27:0/3654150879 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdc1801b9e0 con 0x7fdc280703b0 2023-11-30T21:43:14.476 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.472+0000 7fdc3079c700 1 -- 172.21.15.27:0/3654150879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc280faa00 con 0x7fdc280703b0 2023-11-30T21:43:14.476 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.472+0000 7fdc16ffd700 1 -- 172.21.15.27:0/3654150879 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc1801a8f0 con 0x7fdc280703b0 2023-11-30T21:43:14.476 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.472+0000 7fdc3079c700 1 -- 172.21.15.27:0/3654150879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc280fae10 con 0x7fdc280703b0 2023-11-30T21:43:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.472+0000 7fdc16ffd700 1 -- 172.21.15.27:0/3654150879 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fdc1800f070 con 0x7fdc280703b0 2023-11-30T21:43:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.473+0000 7fdc16ffd700 1 -- 172.21.15.27:0/3654150879 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fdc18025c10 con 0x7fdc280703b0 2023-11-30T21:43:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.473+0000 7fdc3079c700 1 -- 172.21.15.27:0/3654150879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc28045ad0 con 0x7fdc280703b0 2023-11-30T21:43:14.478 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.475+0000 7fdc16ffd700 1 -- 172.21.15.27:0/3654150879 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fdc18016070 con 0x7fdc280703b0 2023-11-30T21:43:14.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 bash[140603]: c26ebacc0897de803e8a9d84c3678108d8c3aa5f0657799d1b7983fb4a749c1b 2023-11-30T21:43:14.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.020+0000 7fab781c5700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-30T21:43:14.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.020+0000 7fab781c5700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-11-30T21:43:14.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.020+0000 7fab781c5700 0 pidfile_write: ignore empty --pid-file 2023-11-30T21:43:14.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 systemd[1]: Started Ceph mon.smithi027 for 6473e080-8fc9-11ee-95a2-87774f69a715. 2023-11-30T21:43:14.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 0 load: jerasure load: lrc load: isa 2023-11-30T21:43:14.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: RocksDB version: 6.8.1 2023-11-30T21:43:14.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-30T21:43:14.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Compile date May 13 2021 2023-11-30T21:43:14.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: DB SUMMARY 2023-11-30T21:43:14.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: CURRENT file: CURRENT 2023-11-30T21:43:14.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: IDENTITY file: IDENTITY 2023-11-30T21:43:14.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-11-30T21:43:14.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi027/store.db dir, Total Num: 1, files: 000004.sst 2023-11-30T21:43:14.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi027/store.db: 000006.log size: 92918 ; 2023-11-30T21:43:14.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.error_if_exists: 0 2023-11-30T21:43:14.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.create_if_missing: 0 2023-11-30T21:43:14.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.paranoid_checks: 1 2023-11-30T21:43:14.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.env: 0x5563887831c0 2023-11-30T21:43:14.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.fs: Posix File System 2023-11-30T21:43:14.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.info_log: 0x55638a14d960 2023-11-30T21:43:14.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-30T21:43:14.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.statistics: (nil) 2023-11-30T21:43:14.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.use_fsync: 0 2023-11-30T21:43:14.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.max_log_file_size: 0 2023-11-30T21:43:14.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-30T21:43:14.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-30T21:43:14.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-30T21:43:14.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-30T21:43:14.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.allow_fallocate: 1 2023-11-30T21:43:14.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-30T21:43:14.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-30T21:43:14.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.use_direct_reads: 0 2023-11-30T21:43:14.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-30T21:43:14.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-30T21:43:14.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.db_log_dir: 2023-11-30T21:43:14.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi027/store.db 2023-11-30T21:43:14.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-30T21:43:14.518 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.max_subcompactions: 1 2023-11-30T21:43:14.518 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.max_background_flushes: -1 2023-11-30T21:43:14.518 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-30T21:43:14.518 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-30T21:43:14.519 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-30T21:43:14.519 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-30T21:43:14.519 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-30T21:43:14.519 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-30T21:43:14.519 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-30T21:43:14.519 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.write_buffer_manager: 0x55638adc2030 2023-11-30T21:43:14.519 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-30T21:43:14.519 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-30T21:43:14.520 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-30T21:43:14.520 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-30T21:43:14.520 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-30T21:43:14.520 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-30T21:43:14.521 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-30T21:43:14.521 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-30T21:43:14.521 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-30T21:43:14.521 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.unordered_write: 0 2023-11-30T21:43:14.522 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-30T21:43:14.522 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-30T21:43:14.522 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-30T21:43:14.522 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-30T21:43:14.523 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.row_cache: None 2023-11-30T21:43:14.523 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.wal_filter: None 2023-11-30T21:43:14.523 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-30T21:43:14.523 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-30T21:43:14.523 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.preserve_deletes: 0 2023-11-30T21:43:14.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.two_write_queues: 0 2023-11-30T21:43:14.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-30T21:43:14.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.atomic_flush: 0 2023-11-30T21:43:14.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-30T21:43:14.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-30T21:43:14.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-30T21:43:14.525 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.log_readahead_size: 0 2023-11-30T21:43:14.525 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-30T21:43:14.525 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.max_background_jobs: 2 2023-11-30T21:43:14.525 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.max_background_compactions: -1 2023-11-30T21:43:14.525 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-30T21:43:14.526 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-30T21:43:14.526 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-30T21:43:14.526 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-30T21:43:14.526 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-30T21:43:14.526 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-30T21:43:14.526 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-30T21:43:14.526 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-30T21:43:14.526 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.max_open_files: -1 2023-11-30T21:43:14.527 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-30T21:43:14.527 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-30T21:43:14.527 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-30T21:43:14.527 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-30T21:43:14.527 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Compression algorithms supported: 2023-11-30T21:43:14.527 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-30T21:43:14.527 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: kZSTD supported: 0 2023-11-30T21:43:14.527 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: kXpressCompression supported: 0 2023-11-30T21:43:14.527 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-30T21:43:14.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: kLZ4Compression supported: 1 2023-11-30T21:43:14.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: kBZip2Compression supported: 0 2023-11-30T21:43:14.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: kZlibCompression supported: 1 2023-11-30T21:43:14.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: kSnappyCompression supported: 1 2023-11-30T21:43:14.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.025+0000 7fab781c5700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-30T21:43:14.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi027/store.db/MANIFEST-000005 2023-11-30T21:43:14.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-30T21:43:14.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-30T21:43:14.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.merge_operator: 2023-11-30T21:43:14.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compaction_filter: None 2023-11-30T21:43:14.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-30T21:43:14.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-30T21:43:14.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-30T21:43:14.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55638a064128) 2023-11-30T21:43:14.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cache_index_and_filter_blocks: 1 2023-11-30T21:43:14.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-30T21:43:14.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-30T21:43:14.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: pin_top_level_index_and_filter: 1 2023-11-30T21:43:14.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: index_type: 0 2023-11-30T21:43:14.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: data_block_index_type: 0 2023-11-30T21:43:14.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: index_shortening: 1 2023-11-30T21:43:14.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: data_block_hash_table_util_ratio: 0.750000 2023-11-30T21:43:14.530 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: hash_index_allow_collision: 1 2023-11-30T21:43:14.530 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: checksum: 1 2023-11-30T21:43:14.530 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: no_block_cache: 0 2023-11-30T21:43:14.530 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: block_cache: 0x55638a09af10 2023-11-30T21:43:14.530 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: block_cache_name: BinnedLRUCache 2023-11-30T21:43:14.530 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: block_cache_options: 2023-11-30T21:43:14.530 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: capacity : 536870912 2023-11-30T21:43:14.530 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: num_shard_bits : 4 2023-11-30T21:43:14.530 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: strict_capacity_limit : 0 2023-11-30T21:43:14.530 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: high_pri_pool_ratio: 0.000 2023-11-30T21:43:14.530 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: block_cache_compressed: (nil) 2023-11-30T21:43:14.531 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: persistent_cache: (nil) 2023-11-30T21:43:14.531 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: block_size: 4096 2023-11-30T21:43:14.531 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: block_size_deviation: 10 2023-11-30T21:43:14.531 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: block_restart_interval: 16 2023-11-30T21:43:14.531 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: index_block_restart_interval: 1 2023-11-30T21:43:14.531 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: metadata_block_size: 4096 2023-11-30T21:43:14.531 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: partition_filters: 0 2023-11-30T21:43:14.531 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: use_delta_encoding: 1 2023-11-30T21:43:14.531 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-30T21:43:14.531 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: whole_key_filtering: 1 2023-11-30T21:43:14.531 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: verify_compression: 0 2023-11-30T21:43:14.531 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: read_amp_bytes_per_bit: 0 2023-11-30T21:43:14.532 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: format_version: 2 2023-11-30T21:43:14.532 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: enable_index_compression: 1 2023-11-30T21:43:14.532 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: block_align: 0 2023-11-30T21:43:14.532 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.532 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-30T21:43:14.532 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-30T21:43:14.532 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compression: NoCompression 2023-11-30T21:43:14.532 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-30T21:43:14.532 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-30T21:43:14.532 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-30T21:43:14.533 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.num_levels: 7 2023-11-30T21:43:14.533 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-30T21:43:14.533 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-30T21:43:14.533 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-30T21:43:14.533 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-30T21:43:14.534 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-30T21:43:14.534 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-30T21:43:14.534 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-30T21:43:14.534 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-30T21:43:14.534 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-30T21:43:14.534 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-30T21:43:14.534 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-30T21:43:14.534 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-30T21:43:14.534 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-30T21:43:14.534 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-30T21:43:14.535 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-30T21:43:14.535 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-30T21:43:14.535 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-30T21:43:14.535 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-30T21:43:14.535 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-30T21:43:14.535 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-30T21:43:14.535 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-30T21:43:14.535 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-30T21:43:14.535 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-30T21:43:14.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-30T21:43:14.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-30T21:43:14.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-30T21:43:14.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-30T21:43:14.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-30T21:43:14.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-30T21:43:14.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-30T21:43:14.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-30T21:43:14.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-30T21:43:14.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-30T21:43:14.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-30T21:43:14.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-30T21:43:14.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-30T21:43:14.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-30T21:43:14.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-30T21:43:14.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-30T21:43:14.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-30T21:43:14.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-30T21:43:14.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-30T21:43:14.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-30T21:43:14.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-30T21:43:14.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-30T21:43:14.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-30T21:43:14.538 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-30T21:43:14.538 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.table_properties_collectors: 2023-11-30T21:43:14.538 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.inplace_update_support: 0 2023-11-30T21:43:14.538 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-30T21:43:14.538 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-30T21:43:14.538 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-30T21:43:14.538 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-30T21:43:14.538 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.bloom_locality: 0 2023-11-30T21:43:14.539 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.max_successive_merges: 0 2023-11-30T21:43:14.539 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-30T21:43:14.539 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-30T21:43:14.539 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-30T21:43:14.539 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-30T21:43:14.539 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.ttl: 2592000 2023-11-30T21:43:14.539 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-30T21:43:14.539 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi027/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-30T21:43:14.539 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.540 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-11-30T21:43:14.540 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.540 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701380594028886, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-11-30T21:43:14.540 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.026+0000 7fab781c5700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-11-30T21:43:14.540 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.027+0000 7fab781c5700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-30T21:43:14.540 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.027+0000 7fab781c5700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701380594029826, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88698, "table_properties": {"data_size": 86813, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79954, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1701380594, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-30T21:43:14.540 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.027+0000 7fab781c5700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-11-30T21:43:14.540 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.540 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.029+0000 7fab781c5700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701380594030252, "job": 1, "event": "recovery_finished"} 2023-11-30T21:43:14.541 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.029+0000 7fab781c5700 4 rocksdb: DB pointer 0x55638a15f800 2023-11-30T21:43:14.541 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.029+0000 7fab5efdd700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-30T21:43:14.541 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.029+0000 7fab5efdd700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-30T21:43:14.541 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: ** DB Stats ** 2023-11-30T21:43:14.541 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-30T21:43:14.541 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-30T21:43:14.541 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-30T21:43:14.541 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-30T21:43:14.541 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-30T21:43:14.541 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-30T21:43:14.541 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-30T21:43:14.541 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.542 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: ** Compaction Stats [default] ** 2023-11-30T21:43:14.542 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-30T21:43:14.542 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-30T21:43:14.542 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.9 0.00 0.00 1 0.001 0 0 2023-11-30T21:43:14.542 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.9 0.00 0.00 1 0.001 0 0 2023-11-30T21:43:14.542 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.9 0.00 0.00 1 0.001 0 0 2023-11-30T21:43:14.542 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.543 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: ** Compaction Stats [default] ** 2023-11-30T21:43:14.543 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-30T21:43:14.543 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-30T21:43:14.543 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 142.9 0.00 0.00 1 0.001 0 0 2023-11-30T21:43:14.544 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-30T21:43:14.544 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-30T21:43:14.544 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-30T21:43:14.544 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: AddFile(Total Files): cumulative 0, interval 0 2023-11-30T21:43:14.544 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-30T21:43:14.544 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: AddFile(Keys): cumulative 0, interval 0 2023-11-30T21:43:14.545 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Cumulative compaction: 0.00 GB write, 27.86 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-30T21:43:14.545 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Interval compaction: 0.00 GB write, 27.86 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-30T21:43:14.545 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-30T21:43:14.545 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.545 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: ** File Read Latency Histogram By Level [default] ** 2023-11-30T21:43:14.546 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.546 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: ** Compaction Stats [default] ** 2023-11-30T21:43:14.546 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-30T21:43:14.547 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-30T21:43:14.547 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: L0 2/0 88.31 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.9 0.00 0.00 1 0.001 0 0 2023-11-30T21:43:14.547 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Sum 2/0 88.31 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.9 0.00 0.00 1 0.001 0 0 2023-11-30T21:43:14.547 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-30T21:43:14.547 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.547 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: ** Compaction Stats [default] ** 2023-11-30T21:43:14.547 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-30T21:43:14.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-30T21:43:14.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 142.9 0.00 0.00 1 0.001 0 0 2023-11-30T21:43:14.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-30T21:43:14.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-30T21:43:14.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-30T21:43:14.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: AddFile(Total Files): cumulative 0, interval 0 2023-11-30T21:43:14.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-30T21:43:14.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: AddFile(Keys): cumulative 0, interval 0 2023-11-30T21:43:14.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Cumulative compaction: 0.00 GB write, 27.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-30T21:43:14.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-30T21:43:14.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-30T21:43:14.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: ** File Read Latency Histogram By Level [default] ** 2023-11-30T21:43:14.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.029+0000 7fab781c5700 0 starting mon.smithi027 rank 0 at public addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] at bind addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi027 fsid 6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:43:14.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.030+0000 7fab781c5700 1 mon.smithi027@-1(???) e1 preinit fsid 6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:43:14.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.030+0000 7fab781c5700 0 mon.smithi027@-1(???).mds e1 new map 2023-11-30T21:43:14.550 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.030+0000 7fab781c5700 0 mon.smithi027@-1(???).mds e1 print_map 2023-11-30T21:43:14.550 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: e1 2023-11-30T21:43:14.550 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: enable_multiple, ever_enabled_multiple: 1,1 2023-11-30T21:43:14.550 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-30T21:43:14.550 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: legacy client fscid: -1 2023-11-30T21:43:14.550 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.550 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: No filesystems configured 2023-11-30T21:43:14.550 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: 2023-11-30T21:43:14.551 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.031+0000 7fab781c5700 0 mon.smithi027@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-30T21:43:14.551 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.031+0000 7fab781c5700 0 mon.smithi027@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-30T21:43:14.551 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.031+0000 7fab781c5700 0 mon.smithi027@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-30T21:43:14.551 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.031+0000 7fab781c5700 0 mon.smithi027@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-30T21:43:14.551 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:12.036133+0000 mon.smithi027 (mon.0) 8 : audit [INF] from='client.? 172.21.15.27:0/1167313741' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-30T21:43:14.552 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:12.039170+0000 mon.smithi027 (mon.0) 9 : audit [INF] from='client.? 172.21.15.27:0/1167313741' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-30T21:43:14.552 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.031+0000 7fab781c5700 1 mon.smithi027@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-30T21:43:14.552 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.032+0000 7fab781c5700 20 expand_channel_meta expand map: {default=false} 2023-11-30T21:43:14.552 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.032+0000 7fab781c5700 20 expand_channel_meta from 'false' to 'false' 2023-11-30T21:43:14.552 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.032+0000 7fab781c5700 20 expand_channel_meta expanded map: {default=false} 2023-11-30T21:43:14.553 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.032+0000 7fab781c5700 20 expand_channel_meta expand map: {default=info} 2023-11-30T21:43:14.553 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.032+0000 7fab781c5700 20 expand_channel_meta from 'info' to 'info' 2023-11-30T21:43:14.553 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.032+0000 7fab781c5700 20 expand_channel_meta expanded map: {default=info} 2023-11-30T21:43:14.553 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.032+0000 7fab781c5700 20 expand_channel_meta expand map: {default=daemon} 2023-11-30T21:43:14.553 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.032+0000 7fab781c5700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-11-30T21:43:14.553 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.032+0000 7fab781c5700 20 expand_channel_meta expanded map: {default=daemon} 2023-11-30T21:43:14.553 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.032+0000 7fab781c5700 20 expand_channel_meta expand map: {default=debug} 2023-11-30T21:43:14.553 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.032+0000 7fab781c5700 20 expand_channel_meta from 'debug' to 'debug' 2023-11-30T21:43:14.554 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:43:14.032+0000 7fab781c5700 20 expand_channel_meta expanded map: {default=debug} 2023-11-30T21:43:14.554 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:14.049659+0000 mon.smithi027 (mon.0) 1 : cluster [INF] mon.smithi027 is new leader, mons smithi027 in quorum (ranks 0) 2023-11-30T21:43:14.554 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:14.049765+0000 mon.smithi027 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi027=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0]} 2023-11-30T21:43:14.554 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:14.049822+0000 mon.smithi027 (mon.0) 3 : cluster [DBG] fsmap 2023-11-30T21:43:14.554 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:14.049865+0000 mon.smithi027 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-30T21:43:14.554 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:14.050473+0000 mon.smithi027 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-11-30T21:43:14.556 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.543+0000 7fdc3079c700 1 -- 172.21.15.27:0/3654150879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fdc28027a00 con 0x7fdc280703b0 2023-11-30T21:43:14.556 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.546+0000 7fdc16ffd700 1 -- 172.21.15.27:0/3654150879 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fdc18025e50 con 0x7fdc280703b0 2023-11-30T21:43:14.556 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.547+0000 7fdc3079c700 1 -- 172.21.15.27:0/3654150879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc280703b0 msgr2=0x7fdc280fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:14.556 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.547+0000 7fdc3079c700 1 --2- 172.21.15.27:0/3654150879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc280703b0 0x7fdc280fc3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fdc1800b190 tx=0x7fdc18006ee0).stop 2023-11-30T21:43:14.556 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.547+0000 7fdc3079c700 1 -- 172.21.15.27:0/3654150879 shutdown_connections 2023-11-30T21:43:14.556 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.547+0000 7fdc3079c700 1 --2- 172.21.15.27:0/3654150879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc280703b0 0x7fdc280fc3e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:14.556 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.547+0000 7fdc3079c700 1 -- 172.21.15.27:0/3654150879 >> 172.21.15.27:0/3654150879 conn(0x7fdc280f63c0 msgr2=0x7fdc280f7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:14.556 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.547+0000 7fdc3079c700 1 -- 172.21.15.27:0/3654150879 shutdown_connections 2023-11-30T21:43:14.556 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:14.547+0000 7fdc3079c700 1 -- 172.21.15.27:0/3654150879 wait complete. 2023-11-30T21:43:14.900 INFO:teuthology.orchestra.run.smithi027.stderr:Wrote config to /etc/ceph/ceph.conf 2023-11-30T21:43:14.900 INFO:teuthology.orchestra.run.smithi027.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-30T21:43:14.900 INFO:teuthology.orchestra.run.smithi027.stderr:Creating mgr... 2023-11-30T21:43:14.901 INFO:teuthology.orchestra.run.smithi027.stderr:Verifying port 9283 ... 2023-11-30T21:43:14.902 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl daemon-reload 2023-11-30T21:43:15.000 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl stop ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mgr.smithi027.brrjcl 2023-11-30T21:43:15.027 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl reset-failed ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mgr.smithi027.brrjcl 2023-11-30T21:43:15.032 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: Failed to reset failed state of unit ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mgr.smithi027.brrjcl.service: Unit ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mgr.smithi027.brrjcl.service not loaded. 2023-11-30T21:43:15.033 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl enable ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mgr.smithi027.brrjcl 2023-11-30T21:43:15.039 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: Created symlink /etc/systemd/system/ceph-6473e080-8fc9-11ee-95a2-87774f69a715.target.wants/ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mgr.smithi027.brrjcl.service → /etc/systemd/system/ceph-6473e080-8fc9-11ee-95a2-87774f69a715@.service. 2023-11-30T21:43:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl start ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mgr.smithi027.brrjcl 2023-11-30T21:43:15.501 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-30T21:43:15.509 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: disabled 2023-11-30T21:43:15.509 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-active firewalld.service 2023-11-30T21:43:15.517 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: inactive 2023-11-30T21:43:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:firewalld.service is not enabled 2023-11-30T21:43:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:Not possible to enable service . firewalld.service is not available 2023-11-30T21:43:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-30T21:43:15.518 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-30T21:43:15.526 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: disabled 2023-11-30T21:43:15.526 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-active firewalld.service 2023-11-30T21:43:15.534 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: inactive 2023-11-30T21:43:15.536 INFO:teuthology.orchestra.run.smithi027.stderr:firewalld.service is not enabled 2023-11-30T21:43:15.536 INFO:teuthology.orchestra.run.smithi027.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-11-30T21:43:15.536 INFO:teuthology.orchestra.run.smithi027.stderr:Waiting for mgr to start... 2023-11-30T21:43:15.536 INFO:teuthology.orchestra.run.smithi027.stderr:Waiting for mgr... 2023-11-30T21:43:15.536 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-30T21:43:15.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:15 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:14.547197+0000 mon.smithi027 (mon.0) 6 : audit [INF] from='client.? 172.21.15.27:0/3654150879' entity='client.admin' 2023-11-30T21:43:15.901 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.898+0000 7fbd9c872700 1 Processor -- start 2023-11-30T21:43:15.901 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.898+0000 7fbd9c872700 1 -- start start 2023-11-30T21:43:15.901 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.898+0000 7fbd9c872700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd980703d0 0x7fbd980fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:15.901 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.898+0000 7fbd9c872700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd980fee30 con 0x7fbd980703d0 2023-11-30T21:43:15.901 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.899+0000 7fbd9659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd980703d0 0x7fbd980fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:15.901 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.899+0000 7fbd9659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd980703d0 0x7fbd980fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45908/0 (socket says 172.21.15.27:45908) 2023-11-30T21:43:15.901 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.899+0000 7fbd9659c700 1 -- 172.21.15.27:0/1232664922 learned_addr learned my addr 172.21.15.27:0/1232664922 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:15.901 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.899+0000 7fbd9659c700 1 -- 172.21.15.27:0/1232664922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd980fef70 con 0x7fbd980703d0 2023-11-30T21:43:15.901 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.899+0000 7fbd9659c700 1 --2- 172.21.15.27:0/1232664922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd980703d0 0x7fbd980fe940 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fbd8800b5f0 tx=0x7fbd88006ee0).ready entity=mon.0 client_cookie=51617d6c695ee5f server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:15.901 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.899+0000 7fbd9559a700 1 -- 172.21.15.27:0/1232664922 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd8800d600 con 0x7fbd980703d0 2023-11-30T21:43:15.902 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.899+0000 7fbd9559a700 1 -- 172.21.15.27:0/1232664922 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbd8800dbd0 con 0x7fbd980703d0 2023-11-30T21:43:15.902 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.899+0000 7fbd9559a700 1 -- 172.21.15.27:0/1232664922 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd88011c00 con 0x7fbd980703d0 2023-11-30T21:43:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.900+0000 7fbd9c872700 1 -- 172.21.15.27:0/1232664922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd980703d0 msgr2=0x7fbd980fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.900+0000 7fbd9c872700 1 --2- 172.21.15.27:0/1232664922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd980703d0 0x7fbd980fe940 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fbd8800b5f0 tx=0x7fbd88006ee0).stop 2023-11-30T21:43:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.900+0000 7fbd9c872700 1 -- 172.21.15.27:0/1232664922 shutdown_connections 2023-11-30T21:43:15.903 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.900+0000 7fbd9c872700 1 --2- 172.21.15.27:0/1232664922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd980703d0 0x7fbd980fe940 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.900+0000 7fbd9c872700 1 -- 172.21.15.27:0/1232664922 >> 172.21.15.27:0/1232664922 conn(0x7fbd980f63c0 msgr2=0x7fbd980f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.901+0000 7fbd9c872700 1 -- 172.21.15.27:0/1232664922 shutdown_connections 2023-11-30T21:43:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.901+0000 7fbd9c872700 1 -- 172.21.15.27:0/1232664922 wait complete. 2023-11-30T21:43:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.901+0000 7fbd9c872700 1 Processor -- start 2023-11-30T21:43:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.901+0000 7fbd9c872700 1 -- start start 2023-11-30T21:43:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.902+0000 7fbd9c872700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd980703d0 0x7fbd9810aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.902+0000 7fbd9c872700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9810af20 con 0x7fbd980703d0 2023-11-30T21:43:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.902+0000 7fbd9659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd980703d0 0x7fbd9810aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.902+0000 7fbd9659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd980703d0 0x7fbd9810aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45918/0 (socket says 172.21.15.27:45918) 2023-11-30T21:43:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.902+0000 7fbd9659c700 1 -- 172.21.15.27:0/2699576455 learned_addr learned my addr 172.21.15.27:0/2699576455 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:15.904 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.902+0000 7fbd9659c700 1 -- 172.21.15.27:0/2699576455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8800b200 con 0x7fbd980703d0 2023-11-30T21:43:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.902+0000 7fbd9659c700 1 --2- 172.21.15.27:0/2699576455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd980703d0 0x7fbd9810aa30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbd8800e040 tx=0x7fbd88006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.902+0000 7fbd86ffd700 1 -- 172.21.15.27:0/2699576455 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd8800d9f0 con 0x7fbd980703d0 2023-11-30T21:43:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.902+0000 7fbd9c872700 1 -- 172.21.15.27:0/2699576455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9810b120 con 0x7fbd980703d0 2023-11-30T21:43:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.902+0000 7fbd86ffd700 1 -- 172.21.15.27:0/2699576455 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbd8801a9f0 con 0x7fbd980703d0 2023-11-30T21:43:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.902+0000 7fbd86ffd700 1 -- 172.21.15.27:0/2699576455 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd8801b8c0 con 0x7fbd980703d0 2023-11-30T21:43:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.902+0000 7fbd9c872700 1 -- 172.21.15.27:0/2699576455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9810b530 con 0x7fbd980703d0 2023-11-30T21:43:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.903+0000 7fbd86ffd700 1 -- 172.21.15.27:0/2699576455 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fbd8800f070 con 0x7fbd980703d0 2023-11-30T21:43:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.903+0000 7fbd86ffd700 1 -- 172.21.15.27:0/2699576455 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fbd88025b30 con 0x7fbd980703d0 2023-11-30T21:43:15.905 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.903+0000 7fbd9c872700 1 -- 172.21.15.27:0/2699576455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd98045ad0 con 0x7fbd980703d0 2023-11-30T21:43:15.906 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.906+0000 7fbd86ffd700 1 -- 172.21.15.27:0/2699576455 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fbd88016070 con 0x7fbd980703d0 2023-11-30T21:43:15.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.976+0000 7fbd9c872700 1 -- 172.21.15.27:0/2699576455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbd98104890 con 0x7fbd980703d0 2023-11-30T21:43:15.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.977+0000 7fbd86ffd700 1 -- 172.21.15.27:0/2699576455 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fbd8801bd20 con 0x7fbd980703d0 2023-11-30T21:43:15.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: { 2023-11-30T21:43:15.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "fsid": "6473e080-8fc9-11ee-95a2-87774f69a715", 2023-11-30T21:43:15.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "health": { 2023-11-30T21:43:15.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-30T21:43:15.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "checks": {}, 2023-11-30T21:43:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "mutes": [] 2023-11-30T21:43:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-30T21:43:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "quorum": [ 2023-11-30T21:43:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 0 2023-11-30T21:43:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: ], 2023-11-30T21:43:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-30T21:43:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "smithi027" 2023-11-30T21:43:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: ], 2023-11-30T21:43:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-11-30T21:43:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "monmap": { 2023-11-30T21:43:15.986 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-30T21:43:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-30T21:43:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-30T21:43:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "osdmap": { 2023-11-30T21:43:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-30T21:43:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-30T21:43:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-30T21:43:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-30T21:43:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-30T21:43:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-30T21:43:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-30T21:43:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:15.987 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "pgmap": { 2023-11-30T21:43:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-30T21:43:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-30T21:43:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-30T21:43:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-30T21:43:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-30T21:43:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-30T21:43:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-30T21:43:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-30T21:43:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "fsmap": { 2023-11-30T21:43:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-30T21:43:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-30T21:43:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-30T21:43:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-30T21:43:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "available": false, 2023-11-30T21:43:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-30T21:43:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "modules": [ 2023-11-30T21:43:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "iostat", 2023-11-30T21:43:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "restful" 2023-11-30T21:43:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: ], 2023-11-30T21:43:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "services": {} 2023-11-30T21:43:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "servicemap": { 2023-11-30T21:43:15.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-30T21:43:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "modified": "2023-11-30T21:43:10.588117+0000", 2023-11-30T21:43:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "services": {} 2023-11-30T21:43:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-30T21:43:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: } 2023-11-30T21:43:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.978+0000 7fbd9c872700 1 -- 172.21.15.27:0/2699576455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd980703d0 msgr2=0x7fbd9810aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.978+0000 7fbd9c872700 1 --2- 172.21.15.27:0/2699576455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd980703d0 0x7fbd9810aa30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbd8800e040 tx=0x7fbd88006ee0).stop 2023-11-30T21:43:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.978+0000 7fbd9c872700 1 -- 172.21.15.27:0/2699576455 shutdown_connections 2023-11-30T21:43:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.978+0000 7fbd9c872700 1 --2- 172.21.15.27:0/2699576455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd980703d0 0x7fbd9810aa30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.978+0000 7fbd9c872700 1 -- 172.21.15.27:0/2699576455 >> 172.21.15.27:0/2699576455 conn(0x7fbd980f63c0 msgr2=0x7fbd980f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.978+0000 7fbd9c872700 1 -- 172.21.15.27:0/2699576455 shutdown_connections 2023-11-30T21:43:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:15.978+0000 7fbd9c872700 1 -- 172.21.15.27:0/2699576455 wait complete. 2023-11-30T21:43:16.400 INFO:teuthology.orchestra.run.smithi027.stderr:mgr not available, waiting (1/15)... 2023-11-30T21:43:16.760 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:16 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:15.978139+0000 mon.smithi027 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.27:0/2699576455' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-30T21:43:18.402 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-30T21:43:18.933 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.930+0000 7f28d8b0d700 1 Processor -- start 2023-11-30T21:43:18.933 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.930+0000 7f28d8b0d700 1 -- start start 2023-11-30T21:43:18.933 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.930+0000 7f28d8b0d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d40fab80 0x7f28d40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:18.933 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.930+0000 7f28d8b0d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40fb490 con 0x7f28d40fab80 2023-11-30T21:43:18.933 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.930+0000 7f28d259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d40fab80 0x7f28d40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:18.933 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.930+0000 7f28d259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d40fab80 0x7f28d40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45930/0 (socket says 172.21.15.27:45930) 2023-11-30T21:43:18.933 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.930+0000 7f28d259c700 1 -- 172.21.15.27:0/2615285980 learned_addr learned my addr 172.21.15.27:0/2615285980 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:18.933 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.931+0000 7f28d259c700 1 -- 172.21.15.27:0/2615285980 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d40fb5d0 con 0x7f28d40fab80 2023-11-30T21:43:18.933 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.931+0000 7f28d259c700 1 --2- 172.21.15.27:0/2615285980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d40fab80 0x7f28d40fafa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f28c400b5f0 tx=0x7f28c4006ee0).ready entity=mon.0 client_cookie=62f2d8525ee62b71 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:18.934 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.931+0000 7f28d159a700 1 -- 172.21.15.27:0/2615285980 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28c400d600 con 0x7f28d40fab80 2023-11-30T21:43:18.934 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.931+0000 7f28d159a700 1 -- 172.21.15.27:0/2615285980 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f28c400dbd0 con 0x7f28d40fab80 2023-11-30T21:43:18.934 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.931+0000 7f28d159a700 1 -- 172.21.15.27:0/2615285980 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28c4011c00 con 0x7f28d40fab80 2023-11-30T21:43:18.934 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.932+0000 7f28d8b0d700 1 -- 172.21.15.27:0/2615285980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d40fab80 msgr2=0x7f28d40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:18.934 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.932+0000 7f28d8b0d700 1 --2- 172.21.15.27:0/2615285980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d40fab80 0x7f28d40fafa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f28c400b5f0 tx=0x7f28c4006ee0).stop 2023-11-30T21:43:18.936 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.932+0000 7f28d8b0d700 1 -- 172.21.15.27:0/2615285980 shutdown_connections 2023-11-30T21:43:18.936 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.932+0000 7f28d8b0d700 1 --2- 172.21.15.27:0/2615285980 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d40fab80 0x7f28d40fafa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:18.936 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.932+0000 7f28d8b0d700 1 -- 172.21.15.27:0/2615285980 >> 172.21.15.27:0/2615285980 conn(0x7f28d40f63c0 msgr2=0x7f28d40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:18.936 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.932+0000 7f28d8b0d700 1 -- 172.21.15.27:0/2615285980 shutdown_connections 2023-11-30T21:43:18.936 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.932+0000 7f28d8b0d700 1 -- 172.21.15.27:0/2615285980 wait complete. 2023-11-30T21:43:18.937 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.933+0000 7f28d8b0d700 1 Processor -- start 2023-11-30T21:43:18.937 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.933+0000 7f28d8b0d700 1 -- start start 2023-11-30T21:43:18.937 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.933+0000 7f28d8b0d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d40fab80 0x7f28d4100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:18.937 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.933+0000 7f28d8b0d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d40ff290 con 0x7f28d40fab80 2023-11-30T21:43:18.937 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.933+0000 7f28d259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d40fab80 0x7f28d4100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:18.937 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.933+0000 7f28d259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d40fab80 0x7f28d4100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45934/0 (socket says 172.21.15.27:45934) 2023-11-30T21:43:18.937 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.933+0000 7f28d259c700 1 -- 172.21.15.27:0/2707581686 learned_addr learned my addr 172.21.15.27:0/2707581686 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:18.937 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.933+0000 7f28d259c700 1 -- 172.21.15.27:0/2707581686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c400b200 con 0x7f28d40fab80 2023-11-30T21:43:18.937 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.933+0000 7f28d259c700 1 --2- 172.21.15.27:0/2707581686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d40fab80 0x7f28d4100e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f28c400e040 tx=0x7f28c4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:18.937 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.934+0000 7f28c2ffd700 1 -- 172.21.15.27:0/2707581686 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28c401b410 con 0x7f28d40fab80 2023-11-30T21:43:18.937 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.934+0000 7f28c2ffd700 1 -- 172.21.15.27:0/2707581686 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f28c401b9e0 con 0x7f28d40fab80 2023-11-30T21:43:18.937 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.934+0000 7f28d8b0d700 1 -- 172.21.15.27:0/2707581686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d40ff430 con 0x7f28d40fab80 2023-11-30T21:43:18.937 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.934+0000 7f28c2ffd700 1 -- 172.21.15.27:0/2707581686 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28c401a820 con 0x7f28d40fab80 2023-11-30T21:43:18.938 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.934+0000 7f28d8b0d700 1 -- 172.21.15.27:0/2707581686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d40ff840 con 0x7f28d40fab80 2023-11-30T21:43:18.938 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.934+0000 7f28c2ffd700 1 -- 172.21.15.27:0/2707581686 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f28c400f070 con 0x7f28d40fab80 2023-11-30T21:43:18.938 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.934+0000 7f28c2ffd700 1 -- 172.21.15.27:0/2707581686 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f28c401ae10 con 0x7f28d40fab80 2023-11-30T21:43:18.938 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.934+0000 7f28d8b0d700 1 -- 172.21.15.27:0/2707581686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d4045ad0 con 0x7f28d40fab80 2023-11-30T21:43:18.939 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:18.938+0000 7f28c2ffd700 1 -- 172.21.15.27:0/2707581686 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f28c402a060 con 0x7f28d40fab80 2023-11-30T21:43:19.018 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:19.008+0000 7f28d8b0d700 1 -- 172.21.15.27:0/2707581686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f28d40008d0 con 0x7f28d40fab80 2023-11-30T21:43:19.018 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:19.008+0000 7f28c2ffd700 1 -- 172.21.15.27:0/2707581686 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f28c4033090 con 0x7f28d40fab80 2023-11-30T21:43:19.018 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:19.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: { 2023-11-30T21:43:19.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "fsid": "6473e080-8fc9-11ee-95a2-87774f69a715", 2023-11-30T21:43:19.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "health": { 2023-11-30T21:43:19.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-30T21:43:19.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "checks": {}, 2023-11-30T21:43:19.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "mutes": [] 2023-11-30T21:43:19.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:19.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-30T21:43:19.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "quorum": [ 2023-11-30T21:43:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 0 2023-11-30T21:43:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: ], 2023-11-30T21:43:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-30T21:43:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "smithi027" 2023-11-30T21:43:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: ], 2023-11-30T21:43:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-11-30T21:43:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "monmap": { 2023-11-30T21:43:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-30T21:43:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-30T21:43:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-30T21:43:19.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "osdmap": { 2023-11-30T21:43:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-30T21:43:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-30T21:43:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-30T21:43:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-30T21:43:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-30T21:43:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-30T21:43:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-30T21:43:19.021 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:19.022 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "pgmap": { 2023-11-30T21:43:19.022 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-30T21:43:19.022 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-30T21:43:19.022 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-30T21:43:19.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-30T21:43:19.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-30T21:43:19.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-30T21:43:19.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-30T21:43:19.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-30T21:43:19.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:19.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "fsmap": { 2023-11-30T21:43:19.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-30T21:43:19.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-30T21:43:19.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-30T21:43:19.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:19.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-30T21:43:19.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "available": false, 2023-11-30T21:43:19.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-30T21:43:19.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "modules": [ 2023-11-30T21:43:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "iostat", 2023-11-30T21:43:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "restful" 2023-11-30T21:43:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: ], 2023-11-30T21:43:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "services": {} 2023-11-30T21:43:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "servicemap": { 2023-11-30T21:43:19.025 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-30T21:43:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "modified": "2023-11-30T21:43:10.588117+0000", 2023-11-30T21:43:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "services": {} 2023-11-30T21:43:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-30T21:43:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: } 2023-11-30T21:43:19.026 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:19.009+0000 7f28d8b0d700 1 -- 172.21.15.27:0/2707581686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d40fab80 msgr2=0x7f28d4100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:19.009+0000 7f28d8b0d700 1 --2- 172.21.15.27:0/2707581686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d40fab80 0x7f28d4100e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f28c400e040 tx=0x7f28c4006ee0).stop 2023-11-30T21:43:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:19.009+0000 7f28d8b0d700 1 -- 172.21.15.27:0/2707581686 shutdown_connections 2023-11-30T21:43:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:19.009+0000 7f28d8b0d700 1 --2- 172.21.15.27:0/2707581686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d40fab80 0x7f28d4100e10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:19.009+0000 7f28d8b0d700 1 -- 172.21.15.27:0/2707581686 >> 172.21.15.27:0/2707581686 conn(0x7f28d40f63c0 msgr2=0x7f28d4070010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:19.009+0000 7f28d8b0d700 1 -- 172.21.15.27:0/2707581686 shutdown_connections 2023-11-30T21:43:19.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:19.009+0000 7f28d8b0d700 1 -- 172.21.15.27:0/2707581686 wait complete. 2023-11-30T21:43:19.350 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:19.009493+0000 mon.smithi027 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.27:0/2707581686' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-30T21:43:19.391 INFO:teuthology.orchestra.run.smithi027.stderr:mgr not available, waiting (2/15)... 2023-11-30T21:43:21.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:20.676820+0000 mon.smithi027 (mon.0) 9 : cluster [INF] Activating manager daemon smithi027.brrjcl 2023-11-30T21:43:21.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:20.679429+0000 mon.smithi027 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi027.brrjcl(active, starting, since 0.0027245s) 2023-11-30T21:43:21.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:20.680395+0000 mon.smithi027 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.27:0/4230820514' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-30T21:43:21.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:20.680512+0000 mon.smithi027 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.27:0/4230820514' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-30T21:43:21.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:20.680622+0000 mon.smithi027 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.27:0/4230820514' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-30T21:43:21.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:20.680727+0000 mon.smithi027 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.27:0/4230820514' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2023-11-30T21:43:21.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:20.681080+0000 mon.smithi027 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.27:0/4230820514' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mgr metadata", "who": "smithi027.brrjcl", "id": "smithi027.brrjcl"}]: dispatch 2023-11-30T21:43:21.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:20.694380+0000 mon.smithi027 (mon.0) 16 : cluster [INF] Manager daemon smithi027.brrjcl is now available 2023-11-30T21:43:21.012 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:20.703801+0000 mon.smithi027 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.27:0/4230820514' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:21.012 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:20.707463+0000 mon.smithi027 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.27:0/4230820514' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/mirror_snapshot_schedule"}]: dispatch 2023-11-30T21:43:21.012 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:20.708729+0000 mon.smithi027 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.27:0/4230820514' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/trash_purge_schedule"}]: dispatch 2023-11-30T21:43:21.012 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:20.713601+0000 mon.smithi027 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.27:0/4230820514' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:21.012 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:20.715715+0000 mon.smithi027 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.27:0/4230820514' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:21.393 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-30T21:43:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.017+0000 7f0e73e69700 1 Processor -- start 2023-11-30T21:43:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.017+0000 7f0e73e69700 1 -- start start 2023-11-30T21:43:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.017+0000 7f0e73e69700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6c070430 0x7f0e6c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.018+0000 7f0e73e69700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c0fedd0 con 0x7f0e6c070430 2023-11-30T21:43:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.018+0000 7f0e71c05700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6c070430 0x7f0e6c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.018+0000 7f0e71c05700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6c070430 0x7f0e6c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45972/0 (socket says 172.21.15.27:45972) 2023-11-30T21:43:22.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.018+0000 7f0e71c05700 1 -- 172.21.15.27:0/1649752804 learned_addr learned my addr 172.21.15.27:0/1649752804 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:22.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.018+0000 7f0e71c05700 1 -- 172.21.15.27:0/1649752804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c0fef10 con 0x7f0e6c070430 2023-11-30T21:43:22.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.018+0000 7f0e71c05700 1 --2- 172.21.15.27:0/1649752804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6c070430 0x7f0e6c0fe8e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f0e5c00b5f0 tx=0x7f0e5c006ee0).ready entity=mon.0 client_cookie=76605284021327d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:22.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.018+0000 7f0e70c03700 1 -- 172.21.15.27:0/1649752804 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e5c00d600 con 0x7f0e6c070430 2023-11-30T21:43:22.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.018+0000 7f0e70c03700 1 -- 172.21.15.27:0/1649752804 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0e5c00dbd0 con 0x7f0e6c070430 2023-11-30T21:43:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.018+0000 7f0e70c03700 1 -- 172.21.15.27:0/1649752804 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e5c011c00 con 0x7f0e6c070430 2023-11-30T21:43:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.019+0000 7f0e73e69700 1 -- 172.21.15.27:0/1649752804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6c070430 msgr2=0x7f0e6c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.019+0000 7f0e73e69700 1 --2- 172.21.15.27:0/1649752804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6c070430 0x7f0e6c0fe8e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f0e5c00b5f0 tx=0x7f0e5c006ee0).stop 2023-11-30T21:43:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.019+0000 7f0e73e69700 1 -- 172.21.15.27:0/1649752804 shutdown_connections 2023-11-30T21:43:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.019+0000 7f0e73e69700 1 --2- 172.21.15.27:0/1649752804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6c070430 0x7f0e6c0fe8e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.019+0000 7f0e73e69700 1 -- 172.21.15.27:0/1649752804 >> 172.21.15.27:0/1649752804 conn(0x7f0e6c0f63c0 msgr2=0x7f0e6c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.019+0000 7f0e73e69700 1 -- 172.21.15.27:0/1649752804 shutdown_connections 2023-11-30T21:43:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.019+0000 7f0e73e69700 1 -- 172.21.15.27:0/1649752804 wait complete. 2023-11-30T21:43:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.020+0000 7f0e73e69700 1 Processor -- start 2023-11-30T21:43:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.020+0000 7f0e73e69700 1 -- start start 2023-11-30T21:43:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.020+0000 7f0e73e69700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6c070430 0x7f0e6c10cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.020+0000 7f0e73e69700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c10d0d0 con 0x7f0e6c070430 2023-11-30T21:43:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.020+0000 7f0e71c05700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6c070430 0x7f0e6c10cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.020+0000 7f0e71c05700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6c070430 0x7f0e6c10cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45982/0 (socket says 172.21.15.27:45982) 2023-11-30T21:43:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.020+0000 7f0e71c05700 1 -- 172.21.15.27:0/1019942145 learned_addr learned my addr 172.21.15.27:0/1019942145 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.020+0000 7f0e71c05700 1 -- 172.21.15.27:0/1019942145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c00b200 con 0x7f0e6c070430 2023-11-30T21:43:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.020+0000 7f0e71c05700 1 --2- 172.21.15.27:0/1019942145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6c070430 0x7f0e6c10cbe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f0e5c00b190 tx=0x7f0e5c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.020+0000 7f0e5a7fc700 1 -- 172.21.15.27:0/1019942145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e5c01b410 con 0x7f0e6c070430 2023-11-30T21:43:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.021+0000 7f0e73e69700 1 -- 172.21.15.27:0/1019942145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e6c10d2d0 con 0x7f0e6c070430 2023-11-30T21:43:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.021+0000 7f0e5a7fc700 1 -- 172.21.15.27:0/1019942145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0e5c01b9e0 con 0x7f0e6c070430 2023-11-30T21:43:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.021+0000 7f0e5a7fc700 1 -- 172.21.15.27:0/1019942145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e5c01a880 con 0x7f0e6c070430 2023-11-30T21:43:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.021+0000 7f0e73e69700 1 -- 172.21.15.27:0/1019942145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6c10d6e0 con 0x7f0e6c070430 2023-11-30T21:43:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.021+0000 7f0e5a7fc700 1 -- 172.21.15.27:0/1019942145 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f0e5c00f070 con 0x7f0e6c070430 2023-11-30T21:43:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.022+0000 7f0e73e69700 1 -- 172.21.15.27:0/1019942145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e6c045ad0 con 0x7f0e6c070430 2023-11-30T21:43:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.022+0000 7f0e5a7fc700 1 --2- 172.21.15.27:0/1019942145 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f0e5402fc50 0x7f0e54032110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.022+0000 7f0e5a7fc700 1 -- 172.21.15.27:0/1019942145 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0e5c0410b0 con 0x7f0e6c070430 2023-11-30T21:43:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.022+0000 7f0e71404700 1 --2- 172.21.15.27:0/1019942145 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f0e5402fc50 0x7f0e54032110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:22.025 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.022+0000 7f0e71404700 1 --2- 172.21.15.27:0/1019942145 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f0e5402fc50 0x7f0e54032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f0e680096a0 tx=0x7f0e68006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:22.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.026+0000 7f0e5a7fc700 1 -- 172.21.15.27:0/1019942145 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e5c0224f0 con 0x7f0e6c070430 2023-11-30T21:43:22.253 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.246+0000 7f0e73e69700 1 -- 172.21.15.27:0/1019942145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0e6c0ffb90 con 0x7f0e6c070430 2023-11-30T21:43:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.246+0000 7f0e5a7fc700 1 -- 172.21.15.27:0/1019942145 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7f0e5c016070 con 0x7f0e6c070430 2023-11-30T21:43:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: { 2023-11-30T21:43:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "fsid": "6473e080-8fc9-11ee-95a2-87774f69a715", 2023-11-30T21:43:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "health": { 2023-11-30T21:43:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-30T21:43:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "checks": {}, 2023-11-30T21:43:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "mutes": [] 2023-11-30T21:43:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-30T21:43:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "quorum": [ 2023-11-30T21:43:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 0 2023-11-30T21:43:22.254 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: ], 2023-11-30T21:43:22.255 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-30T21:43:22.255 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "smithi027" 2023-11-30T21:43:22.255 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: ], 2023-11-30T21:43:22.255 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "quorum_age": 8, 2023-11-30T21:43:22.255 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "monmap": { 2023-11-30T21:43:22.255 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-30T21:43:22.255 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-30T21:43:22.255 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-30T21:43:22.255 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:22.255 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "osdmap": { 2023-11-30T21:43:22.256 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-30T21:43:22.256 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-30T21:43:22.256 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-30T21:43:22.256 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-30T21:43:22.256 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-30T21:43:22.256 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-30T21:43:22.256 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-30T21:43:22.256 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:22.256 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "pgmap": { 2023-11-30T21:43:22.256 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-30T21:43:22.256 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-30T21:43:22.256 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-30T21:43:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-30T21:43:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-30T21:43:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-30T21:43:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-30T21:43:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-30T21:43:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "fsmap": { 2023-11-30T21:43:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-30T21:43:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-30T21:43:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-30T21:43:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-30T21:43:22.257 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "available": true, 2023-11-30T21:43:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-30T21:43:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "modules": [ 2023-11-30T21:43:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "iostat", 2023-11-30T21:43:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "restful" 2023-11-30T21:43:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: ], 2023-11-30T21:43:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "services": {} 2023-11-30T21:43:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "servicemap": { 2023-11-30T21:43:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-30T21:43:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "modified": "2023-11-30T21:43:10.588117+0000", 2023-11-30T21:43:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "services": {} 2023-11-30T21:43:22.258 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: }, 2023-11-30T21:43:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-30T21:43:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: } 2023-11-30T21:43:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.248+0000 7f0e73e69700 1 -- 172.21.15.27:0/1019942145 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f0e5402fc50 msgr2=0x7f0e54032110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.248+0000 7f0e73e69700 1 --2- 172.21.15.27:0/1019942145 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f0e5402fc50 0x7f0e54032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f0e680096a0 tx=0x7f0e68006b40).stop 2023-11-30T21:43:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.248+0000 7f0e73e69700 1 -- 172.21.15.27:0/1019942145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6c070430 msgr2=0x7f0e6c10cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.248+0000 7f0e73e69700 1 --2- 172.21.15.27:0/1019942145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6c070430 0x7f0e6c10cbe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f0e5c00b190 tx=0x7f0e5c006ee0).stop 2023-11-30T21:43:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.248+0000 7f0e73e69700 1 -- 172.21.15.27:0/1019942145 shutdown_connections 2023-11-30T21:43:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.248+0000 7f0e73e69700 1 --2- 172.21.15.27:0/1019942145 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f0e5402fc50 0x7f0e54032110 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.248+0000 7f0e73e69700 1 --2- 172.21.15.27:0/1019942145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e6c070430 0x7f0e6c10cbe0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.248+0000 7f0e73e69700 1 -- 172.21.15.27:0/1019942145 >> 172.21.15.27:0/1019942145 conn(0x7f0e6c0f63c0 msgr2=0x7f0e6c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.248+0000 7f0e73e69700 1 -- 172.21.15.27:0/1019942145 shutdown_connections 2023-11-30T21:43:22.259 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.248+0000 7f0e73e69700 1 -- 172.21.15.27:0/1019942145 wait complete. 2023-11-30T21:43:22.591 INFO:teuthology.orchestra.run.smithi027.stderr:mgr is available 2023-11-30T21:43:22.591 INFO:teuthology.orchestra.run.smithi027.stderr:Enabling cephadm module... 2023-11-30T21:43:22.591 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-11-30T21:43:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.983+0000 7f804cb25700 1 Processor -- start 2023-11-30T21:43:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.983+0000 7f804cb25700 1 -- start start 2023-11-30T21:43:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.983+0000 7f804cb25700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480704a0 0x7f80480fe750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.983+0000 7f804cb25700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80480fec40 con 0x7f80480704a0 2023-11-30T21:43:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.983+0000 7f80477fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480704a0 0x7f80480fe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.983+0000 7f80477fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480704a0 0x7f80480fe750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32832/0 (socket says 172.21.15.27:32832) 2023-11-30T21:43:22.985 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.983+0000 7f80477fe700 1 -- 172.21.15.27:0/3386690203 learned_addr learned my addr 172.21.15.27:0/3386690203 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.983+0000 7f80477fe700 1 -- 172.21.15.27:0/3386690203 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80480fed80 con 0x7f80480704a0 2023-11-30T21:43:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.984+0000 7f80477fe700 1 --2- 172.21.15.27:0/3386690203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480704a0 0x7f80480fe750 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f803800b5f0 tx=0x7f8038006ee0).ready entity=mon.0 client_cookie=7fe679efe909ad0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.984+0000 7f80467fc700 1 -- 172.21.15.27:0/3386690203 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f803800d600 con 0x7f80480704a0 2023-11-30T21:43:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.984+0000 7f80467fc700 1 -- 172.21.15.27:0/3386690203 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f803800dbd0 con 0x7f80480704a0 2023-11-30T21:43:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.984+0000 7f80467fc700 1 -- 172.21.15.27:0/3386690203 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8038011c00 con 0x7f80480704a0 2023-11-30T21:43:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.985+0000 7f804cb25700 1 -- 172.21.15.27:0/3386690203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480704a0 msgr2=0x7f80480fe750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.985+0000 7f804cb25700 1 --2- 172.21.15.27:0/3386690203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480704a0 0x7f80480fe750 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f803800b5f0 tx=0x7f8038006ee0).stop 2023-11-30T21:43:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.985+0000 7f804cb25700 1 -- 172.21.15.27:0/3386690203 shutdown_connections 2023-11-30T21:43:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.985+0000 7f804cb25700 1 --2- 172.21.15.27:0/3386690203 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480704a0 0x7f80480fe750 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.985+0000 7f804cb25700 1 -- 172.21.15.27:0/3386690203 >> 172.21.15.27:0/3386690203 conn(0x7f80480f6250 msgr2=0x7f80480f8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.985+0000 7f804cb25700 1 -- 172.21.15.27:0/3386690203 shutdown_connections 2023-11-30T21:43:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.985+0000 7f804cb25700 1 -- 172.21.15.27:0/3386690203 wait complete. 2023-11-30T21:43:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.985+0000 7f804cb25700 1 Processor -- start 2023-11-30T21:43:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.985+0000 7f804cb25700 1 -- start start 2023-11-30T21:43:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.986+0000 7f804cb25700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480704a0 0x7f804810a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.986+0000 7f804cb25700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804810ada0 con 0x7f80480704a0 2023-11-30T21:43:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.986+0000 7f80477fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480704a0 0x7f804810a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.986+0000 7f80477fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480704a0 0x7f804810a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32840/0 (socket says 172.21.15.27:32840) 2023-11-30T21:43:22.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.986+0000 7f80477fe700 1 -- 172.21.15.27:0/2126344255 learned_addr learned my addr 172.21.15.27:0/2126344255 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:22.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.986+0000 7f80477fe700 1 -- 172.21.15.27:0/2126344255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803800b200 con 0x7f80480704a0 2023-11-30T21:43:22.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.986+0000 7f80477fe700 1 --2- 172.21.15.27:0/2126344255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480704a0 0x7f804810a8b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f803800b1d0 tx=0x7f8038006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:22.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.986+0000 7f802ffff700 1 -- 172.21.15.27:0/2126344255 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8038011a20 con 0x7f80480704a0 2023-11-30T21:43:22.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.986+0000 7f802ffff700 1 -- 172.21.15.27:0/2126344255 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f803801b880 con 0x7f80480704a0 2023-11-30T21:43:22.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.986+0000 7f804cb25700 1 -- 172.21.15.27:0/2126344255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f804810afa0 con 0x7f80480704a0 2023-11-30T21:43:22.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.986+0000 7f802ffff700 1 -- 172.21.15.27:0/2126344255 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f803801a700 con 0x7f80480704a0 2023-11-30T21:43:22.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.986+0000 7f804cb25700 1 -- 172.21.15.27:0/2126344255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f804810b3b0 con 0x7f80480704a0 2023-11-30T21:43:22.990 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.987+0000 7f804cb25700 1 -- 172.21.15.27:0/2126344255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8048045ad0 con 0x7f80480704a0 2023-11-30T21:43:22.991 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.988+0000 7f802ffff700 1 -- 172.21.15.27:0/2126344255 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f803800f070 con 0x7f80480704a0 2023-11-30T21:43:22.991 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.988+0000 7f802ffff700 1 --2- 172.21.15.27:0/2126344255 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f803402fc70 0x7f8034032130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:22.991 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.988+0000 7f802ffff700 1 -- 172.21.15.27:0/2126344255 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f80380417f0 con 0x7f80480704a0 2023-11-30T21:43:22.991 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.988+0000 7f8046ffd700 1 --2- 172.21.15.27:0/2126344255 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f803402fc70 0x7f8034032130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:22.991 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.988+0000 7f8046ffd700 1 --2- 172.21.15.27:0/2126344255 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f803402fc70 0x7f8034032130 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f803c0096a0 tx=0x7f803c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:22.993 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:22.992+0000 7f802ffff700 1 -- 172.21.15.27:0/2126344255 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f803802d980 con 0x7f80480704a0 2023-11-30T21:43:23.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:22 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:21.683280+0000 mon.smithi027 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi027.brrjcl(active, since 1.00657s) 2023-11-30T21:43:23.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:22 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:22.247635+0000 mon.smithi027 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.27:0/1019942145' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-30T21:43:23.188 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:23.187+0000 7f804cb25700 1 -- 172.21.15.27:0/2126344255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f80480ffa00 con 0x7f80480704a0 2023-11-30T21:43:23.681 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:23.680+0000 7f802ffff700 1 -- 172.21.15.27:0/2126344255 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f803801b410 con 0x7f80480704a0 2023-11-30T21:43:23.682 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:23.682+0000 7f802ffff700 1 -- 172.21.15.27:0/2126344255 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v4) v1 ==== 86+0+0 (secure 0 0 0) 0x7f8038025cb0 con 0x7f80480704a0 2023-11-30T21:43:23.685 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:23.684+0000 7f804cb25700 1 -- 172.21.15.27:0/2126344255 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f803402fc70 msgr2=0x7f8034032130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:23.686 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:23.684+0000 7f804cb25700 1 --2- 172.21.15.27:0/2126344255 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f803402fc70 0x7f8034032130 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f803c0096a0 tx=0x7f803c006b40).stop 2023-11-30T21:43:23.686 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:23.684+0000 7f804cb25700 1 -- 172.21.15.27:0/2126344255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480704a0 msgr2=0x7f804810a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:23.686 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:23.684+0000 7f804cb25700 1 --2- 172.21.15.27:0/2126344255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480704a0 0x7f804810a8b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f803800b1d0 tx=0x7f8038006ee0).stop 2023-11-30T21:43:23.686 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:23.684+0000 7f804cb25700 1 -- 172.21.15.27:0/2126344255 shutdown_connections 2023-11-30T21:43:23.686 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:23.684+0000 7f804cb25700 1 --2- 172.21.15.27:0/2126344255 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f803402fc70 0x7f8034032130 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:23.686 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:23.684+0000 7f804cb25700 1 --2- 172.21.15.27:0/2126344255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80480704a0 0x7f804810a8b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:23.687 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:23.684+0000 7f804cb25700 1 -- 172.21.15.27:0/2126344255 >> 172.21.15.27:0/2126344255 conn(0x7f80480f6250 msgr2=0x7f80480f7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:23.687 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:23.685+0000 7f804cb25700 1 -- 172.21.15.27:0/2126344255 shutdown_connections 2023-11-30T21:43:23.687 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:23.685+0000 7f804cb25700 1 -- 172.21.15.27:0/2126344255 wait complete. 2023-11-30T21:43:23.991 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:23 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:23.188631+0000 mon.smithi027 (mon.0) 24 : audit [INF] from='client.? 172.21.15.27:0/2126344255' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-30T21:43:24.115 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-30T21:43:24.489 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.481+0000 7f1a9f617700 1 Processor -- start 2023-11-30T21:43:24.489 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.482+0000 7f1a9f617700 1 -- start start 2023-11-30T21:43:24.489 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.482+0000 7f1a9f617700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a980703d0 0x7f1a98100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:24.489 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.482+0000 7f1a9f617700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a980fc730 con 0x7f1a980703d0 2023-11-30T21:43:24.489 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.482+0000 7f1a9d3b3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a980703d0 0x7f1a98100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:24.490 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.482+0000 7f1a9d3b3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a980703d0 0x7f1a98100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32864/0 (socket says 172.21.15.27:32864) 2023-11-30T21:43:24.490 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.482+0000 7f1a9d3b3700 1 -- 172.21.15.27:0/3618924447 learned_addr learned my addr 172.21.15.27:0/3618924447 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:24.490 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.483+0000 7f1a9d3b3700 1 -- 172.21.15.27:0/3618924447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a98101230 con 0x7f1a980703d0 2023-11-30T21:43:24.490 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.483+0000 7f1a9d3b3700 1 --2- 172.21.15.27:0/3618924447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a980703d0 0x7f1a98100d40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f1a9400b5f0 tx=0x7f1a94006ee0).ready entity=mon.0 client_cookie=f5226b88f8fdfe2f server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:24.491 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.483+0000 7f1a87fff700 1 -- 172.21.15.27:0/3618924447 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9400d600 con 0x7f1a980703d0 2023-11-30T21:43:24.491 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.483+0000 7f1a87fff700 1 -- 172.21.15.27:0/3618924447 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1a9400dbd0 con 0x7f1a980703d0 2023-11-30T21:43:24.491 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.483+0000 7f1a87fff700 1 -- 172.21.15.27:0/3618924447 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a94011c00 con 0x7f1a980703d0 2023-11-30T21:43:24.491 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.484+0000 7f1a9f617700 1 -- 172.21.15.27:0/3618924447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a980703d0 msgr2=0x7f1a98100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:24.492 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.484+0000 7f1a9f617700 1 --2- 172.21.15.27:0/3618924447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a980703d0 0x7f1a98100d40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f1a9400b5f0 tx=0x7f1a94006ee0).stop 2023-11-30T21:43:24.492 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.484+0000 7f1a9f617700 1 -- 172.21.15.27:0/3618924447 shutdown_connections 2023-11-30T21:43:24.492 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.484+0000 7f1a9f617700 1 --2- 172.21.15.27:0/3618924447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a980703d0 0x7f1a98100d40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:24.492 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.484+0000 7f1a9f617700 1 -- 172.21.15.27:0/3618924447 >> 172.21.15.27:0/3618924447 conn(0x7f1a980f63c0 msgr2=0x7f1a980f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:24.492 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.484+0000 7f1a9f617700 1 -- 172.21.15.27:0/3618924447 shutdown_connections 2023-11-30T21:43:24.492 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.484+0000 7f1a9f617700 1 -- 172.21.15.27:0/3618924447 wait complete. 2023-11-30T21:43:24.493 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.485+0000 7f1a9f617700 1 Processor -- start 2023-11-30T21:43:24.493 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.485+0000 7f1a9f617700 1 -- start start 2023-11-30T21:43:24.493 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.485+0000 7f1a9f617700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a980703d0 0x7f1a9810cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:24.493 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.485+0000 7f1a9f617700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9810d0a0 con 0x7f1a980703d0 2023-11-30T21:43:24.494 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.485+0000 7f1a9d3b3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a980703d0 0x7f1a9810cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:24.494 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.485+0000 7f1a9d3b3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a980703d0 0x7f1a9810cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32878/0 (socket says 172.21.15.27:32878) 2023-11-30T21:43:24.494 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.485+0000 7f1a9d3b3700 1 -- 172.21.15.27:0/566306061 learned_addr learned my addr 172.21.15.27:0/566306061 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:24.494 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.485+0000 7f1a9d3b3700 1 -- 172.21.15.27:0/566306061 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9400b200 con 0x7f1a980703d0 2023-11-30T21:43:24.494 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.486+0000 7f1a9d3b3700 1 --2- 172.21.15.27:0/566306061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a980703d0 0x7f1a9810cbb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f1a9400b190 tx=0x7f1a94006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:24.495 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.486+0000 7f1a85ffb700 1 -- 172.21.15.27:0/566306061 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9401b410 con 0x7f1a980703d0 2023-11-30T21:43:24.495 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.486+0000 7f1a9f617700 1 -- 172.21.15.27:0/566306061 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a9810d2a0 con 0x7f1a980703d0 2023-11-30T21:43:24.495 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.486+0000 7f1a85ffb700 1 -- 172.21.15.27:0/566306061 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1a9401b9e0 con 0x7f1a980703d0 2023-11-30T21:43:24.495 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.486+0000 7f1a85ffb700 1 -- 172.21.15.27:0/566306061 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9401a880 con 0x7f1a980703d0 2023-11-30T21:43:24.495 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.486+0000 7f1a9f617700 1 -- 172.21.15.27:0/566306061 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a9810d6b0 con 0x7f1a980703d0 2023-11-30T21:43:24.495 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.487+0000 7f1a9f617700 1 -- 172.21.15.27:0/566306061 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a98045ad0 con 0x7f1a980703d0 2023-11-30T21:43:24.495 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.487+0000 7f1a85ffb700 1 -- 172.21.15.27:0/566306061 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f1a9400f070 con 0x7f1a980703d0 2023-11-30T21:43:24.495 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.487+0000 7f1a85ffb700 1 --2- 172.21.15.27:0/566306061 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f1a8002fd50 0x7f1a80032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:24.495 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.487+0000 7f1a85ffb700 1 -- 172.21.15.27:0/566306061 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1a940411f0 con 0x7f1a980703d0 2023-11-30T21:43:24.495 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.487+0000 7f1a9cbb2700 1 -- 172.21.15.27:0/566306061 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f1a8002fd50 msgr2=0x7f1a80032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1484297859 2023-11-30T21:43:24.495 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.487+0000 7f1a9cbb2700 1 --2- 172.21.15.27:0/566306061 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f1a8002fd50 0x7f1a80032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-30T21:43:24.496 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.495+0000 7f1a85ffb700 1 -- 172.21.15.27:0/566306061 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a94019cd0 con 0x7f1a980703d0 2023-11-30T21:43:24.689 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.688+0000 7f1a9cbb2700 1 -- 172.21.15.27:0/566306061 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f1a8002fd50 msgr2=0x7f1a80032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1484297859 2023-11-30T21:43:24.690 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.688+0000 7f1a9cbb2700 1 --2- 172.21.15.27:0/566306061 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f1a8002fd50 0x7f1a80032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-30T21:43:24.694 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.694+0000 7f1a9f617700 1 -- 172.21.15.27:0/566306061 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f1a9810e7c0 con 0x7f1a980703d0 2023-11-30T21:43:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.694+0000 7f1a85ffb700 1 -- 172.21.15.27:0/566306061 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v4) v1 ==== 56+0+103 (secure 0 0 0) 0x7f1a94016070 con 0x7f1a980703d0 2023-11-30T21:43:24.695 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: { 2023-11-30T21:43:24.696 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "epoch": 4, 2023-11-30T21:43:24.696 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "available": true, 2023-11-30T21:43:24.696 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "active_name": "smithi027.brrjcl", 2023-11-30T21:43:24.696 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-30T21:43:24.696 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: } 2023-11-30T21:43:24.698 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.696+0000 7f1a9f617700 1 -- 172.21.15.27:0/566306061 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f1a8002fd50 msgr2=0x7f1a80032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-30T21:43:24.698 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.696+0000 7f1a9f617700 1 --2- 172.21.15.27:0/566306061 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f1a8002fd50 0x7f1a80032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-30T21:43:24.698 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.696+0000 7f1a9f617700 1 -- 172.21.15.27:0/566306061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a980703d0 msgr2=0x7f1a9810cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:24.698 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.696+0000 7f1a9f617700 1 --2- 172.21.15.27:0/566306061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a980703d0 0x7f1a9810cbb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f1a9400b190 tx=0x7f1a94006ee0).stop 2023-11-30T21:43:24.699 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.696+0000 7f1a9f617700 1 -- 172.21.15.27:0/566306061 shutdown_connections 2023-11-30T21:43:24.699 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.696+0000 7f1a9f617700 1 --2- 172.21.15.27:0/566306061 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f1a8002fd50 0x7f1a80032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-30T21:43:24.699 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.696+0000 7f1a9f617700 1 --2- 172.21.15.27:0/566306061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a980703d0 0x7f1a9810cbb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:24.699 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.696+0000 7f1a9f617700 1 -- 172.21.15.27:0/566306061 >> 172.21.15.27:0/566306061 conn(0x7f1a980f63c0 msgr2=0x7f1a980f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:24.699 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.696+0000 7f1a9f617700 1 -- 172.21.15.27:0/566306061 shutdown_connections 2023-11-30T21:43:24.699 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:24.696+0000 7f1a9f617700 1 -- 172.21.15.27:0/566306061 wait complete. 2023-11-30T21:43:25.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:24 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:23.682906+0000 mon.smithi027 (mon.0) 25 : audit [INF] from='client.? 172.21.15.27:0/2126344255' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-30T21:43:25.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:24 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:23.682976+0000 mon.smithi027 (mon.0) 26 : cluster [DBG] mgrmap e4: smithi027.brrjcl(active, since 3s) 2023-11-30T21:43:25.048 INFO:teuthology.orchestra.run.smithi027.stderr:Waiting for the mgr to restart... 2023-11-30T21:43:25.048 INFO:teuthology.orchestra.run.smithi027.stderr:Waiting for mgr epoch 4... 2023-11-30T21:43:25.048 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-30T21:43:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.574+0000 7f311231a700 1 Processor -- start 2023-11-30T21:43:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.574+0000 7f311231a700 1 -- start start 2023-11-30T21:43:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.575+0000 7f311231a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f310c0703f0 0x7f310c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.575+0000 7f311231a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c0fee10 con 0x7f310c0703f0 2023-11-30T21:43:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.575+0000 7f310bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f310c0703f0 0x7f310c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.575+0000 7f310bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f310c0703f0 0x7f310c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32880/0 (socket says 172.21.15.27:32880) 2023-11-30T21:43:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.575+0000 7f310bfff700 1 -- 172.21.15.27:0/2414549516 learned_addr learned my addr 172.21.15.27:0/2414549516 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.575+0000 7f310bfff700 1 -- 172.21.15.27:0/2414549516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c0fef50 con 0x7f310c0703f0 2023-11-30T21:43:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.575+0000 7f310bfff700 1 --2- 172.21.15.27:0/2414549516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f310c0703f0 0x7f310c0fe920 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f30fc00b5f0 tx=0x7f30fc006ee0).ready entity=mon.0 client_cookie=976b44a7a9518f server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.576+0000 7f310affd700 1 -- 172.21.15.27:0/2414549516 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30fc00d600 con 0x7f310c0703f0 2023-11-30T21:43:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.576+0000 7f310affd700 1 -- 172.21.15.27:0/2414549516 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f30fc00dbd0 con 0x7f310c0703f0 2023-11-30T21:43:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.576+0000 7f310affd700 1 -- 172.21.15.27:0/2414549516 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30fc011c00 con 0x7f310c0703f0 2023-11-30T21:43:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.577+0000 7f311231a700 1 -- 172.21.15.27:0/2414549516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f310c0703f0 msgr2=0x7f310c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.577+0000 7f311231a700 1 --2- 172.21.15.27:0/2414549516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f310c0703f0 0x7f310c0fe920 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f30fc00b5f0 tx=0x7f30fc006ee0).stop 2023-11-30T21:43:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.577+0000 7f311231a700 1 -- 172.21.15.27:0/2414549516 shutdown_connections 2023-11-30T21:43:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.577+0000 7f311231a700 1 --2- 172.21.15.27:0/2414549516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f310c0703f0 0x7f310c0fe920 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.577+0000 7f311231a700 1 -- 172.21.15.27:0/2414549516 >> 172.21.15.27:0/2414549516 conn(0x7f310c0f63c0 msgr2=0x7f310c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.577+0000 7f311231a700 1 -- 172.21.15.27:0/2414549516 shutdown_connections 2023-11-30T21:43:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.577+0000 7f311231a700 1 -- 172.21.15.27:0/2414549516 wait complete. 2023-11-30T21:43:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.578+0000 7f311231a700 1 Processor -- start 2023-11-30T21:43:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.578+0000 7f311231a700 1 -- start start 2023-11-30T21:43:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.578+0000 7f311231a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f310c0703f0 0x7f310c10a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.578+0000 7f311231a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c10aeb0 con 0x7f310c0703f0 2023-11-30T21:43:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.578+0000 7f310bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f310c0703f0 0x7f310c10a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.578+0000 7f310bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f310c0703f0 0x7f310c10a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32890/0 (socket says 172.21.15.27:32890) 2023-11-30T21:43:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.578+0000 7f310bfff700 1 -- 172.21.15.27:0/1084134834 learned_addr learned my addr 172.21.15.27:0/1084134834 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.579+0000 7f310bfff700 1 -- 172.21.15.27:0/1084134834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc00b200 con 0x7f310c0703f0 2023-11-30T21:43:25.586 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.579+0000 7f310bfff700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f310c0703f0 0x7f310c10a9c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f30fc00b190 tx=0x7f30fc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:25.586 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.579+0000 7f3108ff9700 1 -- 172.21.15.27:0/1084134834 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30fc01b410 con 0x7f310c0703f0 2023-11-30T21:43:25.586 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.579+0000 7f3108ff9700 1 -- 172.21.15.27:0/1084134834 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f30fc01b9e0 con 0x7f310c0703f0 2023-11-30T21:43:25.586 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.579+0000 7f3108ff9700 1 -- 172.21.15.27:0/1084134834 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30fc01a880 con 0x7f310c0703f0 2023-11-30T21:43:25.586 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.579+0000 7f311231a700 1 -- 172.21.15.27:0/1084134834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f310c10b0b0 con 0x7f310c0703f0 2023-11-30T21:43:25.586 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.579+0000 7f311231a700 1 -- 172.21.15.27:0/1084134834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310c10b4c0 con 0x7f310c0703f0 2023-11-30T21:43:25.586 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.580+0000 7f3108ff9700 1 -- 172.21.15.27:0/1084134834 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f30fc00f070 con 0x7f310c0703f0 2023-11-30T21:43:25.586 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.580+0000 7f3108ff9700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f30f802fdb0 0x7f30f8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:25.586 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.580+0000 7f3108ff9700 1 -- 172.21.15.27:0/1084134834 --> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30f8032930 con 0x7f30f802fdb0 2023-11-30T21:43:25.586 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.580+0000 7f3108ff9700 1 -- 172.21.15.27:0/1084134834 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f30fc0411a0 con 0x7f310c0703f0 2023-11-30T21:43:25.586 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.580+0000 7f310b7fe700 1 -- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f30f802fdb0 msgr2=0x7f30f8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1484297859 2023-11-30T21:43:25.587 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.580+0000 7f310b7fe700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f30f802fdb0 0x7f30f8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-30T21:43:25.782 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.781+0000 7f310b7fe700 1 -- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f30f802fdb0 msgr2=0x7f30f8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1484297859 2023-11-30T21:43:25.782 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:25.781+0000 7f310b7fe700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f30f802fdb0 0x7f30f8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-30T21:43:26.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:24.695135+0000 mon.smithi027 (mon.0) 27 : audit [DBG] from='client.? 172.21.15.27:0/566306061' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-30T21:43:26.184 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:26.182+0000 7f310b7fe700 1 -- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f30f802fdb0 msgr2=0x7f30f8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1484297859 2023-11-30T21:43:26.185 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:26.182+0000 7f310b7fe700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f30f802fdb0 0x7f30f8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-30T21:43:26.984 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:26.983+0000 7f310b7fe700 1 -- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f30f802fdb0 msgr2=0x7f30f8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1484297859 2023-11-30T21:43:26.984 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:26.983+0000 7f310b7fe700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f30f802fdb0 0x7f30f8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-30T21:43:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:28.584+0000 7f310b7fe700 1 -- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f30f802fdb0 msgr2=0x7f30f8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1484297859 2023-11-30T21:43:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:28.584+0000 7f310b7fe700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f30f802fdb0 0x7f30f8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-30T21:43:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:28.931+0000 7f3108ff9700 1 -- 172.21.15.27:0/1084134834 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mgrmap(e 5) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f30fc01a9e0 con 0x7f310c0703f0 2023-11-30T21:43:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:28.931+0000 7f3108ff9700 1 -- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f30f802fdb0 msgr2=0x7f30f8032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-30T21:43:28.933 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:28.931+0000 7f3108ff9700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f30f802fdb0 0x7f30f8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-30T21:43:29.260 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:28 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:28.880630+0000 mon.smithi027 (mon.0) 28 : cluster [INF] Active manager daemon smithi027.brrjcl restarted 2023-11-30T21:43:29.261 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:28 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:28.881031+0000 mon.smithi027 (mon.0) 29 : cluster [INF] Activating manager daemon smithi027.brrjcl 2023-11-30T21:43:29.261 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:28 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:28.882986+0000 mon.smithi027 (mon.0) 30 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-11-30T21:43:29.942 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.940+0000 7f3108ff9700 1 -- 172.21.15.27:0/1084134834 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f30fc042a00 con 0x7f310c0703f0 2023-11-30T21:43:29.943 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.941+0000 7f3108ff9700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f30f8033880 0x7f30f8035c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:29.943 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.941+0000 7f3108ff9700 1 -- 172.21.15.27:0/1084134834 --> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f30f8032930 con 0x7f30f8033880 2023-11-30T21:43:29.943 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.941+0000 7f310b7fe700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f30f8033880 0x7f30f8035c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:29.943 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.942+0000 7f310b7fe700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f30f8033880 0x7f30f8035c70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f3100006fd0 tx=0x7f3100006d70).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:29.943 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.943+0000 7f3108ff9700 1 -- 172.21.15.27:0/1084134834 <== mgr.14114 v2:172.21.15.27:6800/2518541463 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f30f8032930 con 0x7f30f8033880 2023-11-30T21:43:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.949+0000 7f311231a700 1 -- 172.21.15.27:0/1084134834 --> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f310c059100 con 0x7f30f8033880 2023-11-30T21:43:29.950 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.949+0000 7f3108ff9700 1 -- 172.21.15.27:0/1084134834 <== mgr.14114 v2:172.21.15.27:6800/2518541463 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f310c059100 con 0x7f30f8033880 2023-11-30T21:43:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: { 2023-11-30T21:43:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "mgrmap_epoch": 6, 2023-11-30T21:43:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "initialized": true 2023-11-30T21:43:29.951 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: } 2023-11-30T21:43:29.952 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.950+0000 7f311231a700 1 -- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f30f8033880 msgr2=0x7f30f8035c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:29.953 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.950+0000 7f311231a700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f30f8033880 0x7f30f8035c70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f3100006fd0 tx=0x7f3100006d70).stop 2023-11-30T21:43:29.953 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.950+0000 7f311231a700 1 -- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f310c0703f0 msgr2=0x7f310c10a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:29.953 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.950+0000 7f311231a700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f310c0703f0 0x7f310c10a9c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f30fc00b190 tx=0x7f30fc006ee0).stop 2023-11-30T21:43:29.953 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.950+0000 7f311231a700 1 -- 172.21.15.27:0/1084134834 shutdown_connections 2023-11-30T21:43:29.953 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.950+0000 7f311231a700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f30f8033880 0x7f30f8035c70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:29.953 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.950+0000 7f311231a700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f310c0703f0 0x7f310c10a9c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:29.953 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.950+0000 7f311231a700 1 --2- 172.21.15.27:0/1084134834 >> [v2:172.21.15.27:6800/1484297859,v1:172.21.15.27:6801/1484297859] conn(0x7f30f802fdb0 0x7f30f8032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-30T21:43:29.953 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.950+0000 7f311231a700 1 -- 172.21.15.27:0/1084134834 >> 172.21.15.27:0/1084134834 conn(0x7f310c0f63c0 msgr2=0x7f310c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:29.954 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.950+0000 7f311231a700 1 -- 172.21.15.27:0/1084134834 shutdown_connections 2023-11-30T21:43:29.954 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:29.950+0000 7f311231a700 1 -- 172.21.15.27:0/1084134834 wait complete. 2023-11-30T21:43:30.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:28.933487+0000 mon.smithi027 (mon.0) 31 : cluster [DBG] mgrmap e5: smithi027.brrjcl(active, starting, since 0.0525262s) 2023-11-30T21:43:30.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:28.934368+0000 mon.smithi027 (mon.0) 32 : audit [DBG] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2023-11-30T21:43:30.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:28.934986+0000 mon.smithi027 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mgr metadata", "who": "smithi027.brrjcl", "id": "smithi027.brrjcl"}]: dispatch 2023-11-30T21:43:30.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:28.935712+0000 mon.smithi027 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-30T21:43:30.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:28.935926+0000 mon.smithi027 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-30T21:43:30.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:28.936107+0000 mon.smithi027 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-30T21:43:30.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:28.950244+0000 mon.smithi027 (mon.0) 37 : cluster [INF] Manager daemon smithi027.brrjcl is now available 2023-11-30T21:43:30.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:28.966719+0000 mon.smithi027 (mon.0) 38 : audit [INF] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:30.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:28.972236+0000 mon.smithi027 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:30.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:28.977325+0000 mon.smithi027 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:30.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:29.074060+0000 mon.smithi027 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:30.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:29.075078+0000 mon.smithi027 (mon.0) 42 : audit [DBG] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-30T21:43:30.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:29.076476+0000 mon.smithi027 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-30T21:43:30.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:29.080980+0000 mon.smithi027 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-30T21:43:30.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:29.087569+0000 mon.smithi027 (mon.0) 45 : audit [INF] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/mirror_snapshot_schedule"}]: dispatch 2023-11-30T21:43:30.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:29.088733+0000 mon.smithi027 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/trash_purge_schedule"}]: dispatch 2023-11-30T21:43:30.326 INFO:teuthology.orchestra.run.smithi027.stderr:mgr epoch 4 is available 2023-11-30T21:43:30.326 INFO:teuthology.orchestra.run.smithi027.stderr:Setting orchestrator backend to cephadm... 2023-11-30T21:43:30.326 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-11-30T21:43:30.670 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.668+0000 7fe06f59e700 1 Processor -- start 2023-11-30T21:43:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.668+0000 7fe06f59e700 1 -- start start 2023-11-30T21:43:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.668+0000 7fe06f59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0700fa9d0 0x7fe0700fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:30.671 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.668+0000 7fe06f59e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0700fb2e0 con 0x7fe0700fa9d0 2023-11-30T21:43:30.672 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.668+0000 7fe06e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0700fa9d0 0x7fe0700fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:30.672 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.668+0000 7fe06e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0700fa9d0 0x7fe0700fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32936/0 (socket says 172.21.15.27:32936) 2023-11-30T21:43:30.672 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.668+0000 7fe06e59c700 1 -- 172.21.15.27:0/1451083157 learned_addr learned my addr 172.21.15.27:0/1451083157 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:30.672 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.669+0000 7fe06e59c700 1 -- 172.21.15.27:0/1451083157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0700fb420 con 0x7fe0700fa9d0 2023-11-30T21:43:30.674 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.669+0000 7fe06e59c700 1 --2- 172.21.15.27:0/1451083157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0700fa9d0 0x7fe0700fadf0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fe06000b5f0 tx=0x7fe060006ee0).ready entity=mon.0 client_cookie=fb0b842ef5fc163b server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:30.674 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.669+0000 7fe06d59a700 1 -- 172.21.15.27:0/1451083157 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe06000d600 con 0x7fe0700fa9d0 2023-11-30T21:43:30.674 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.669+0000 7fe06d59a700 1 -- 172.21.15.27:0/1451083157 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe06000dbd0 con 0x7fe0700fa9d0 2023-11-30T21:43:30.674 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.669+0000 7fe06d59a700 1 -- 172.21.15.27:0/1451083157 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe060011c00 con 0x7fe0700fa9d0 2023-11-30T21:43:30.674 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.669+0000 7fe06f59e700 1 -- 172.21.15.27:0/1451083157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0700fa9d0 msgr2=0x7fe0700fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:30.674 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.669+0000 7fe06f59e700 1 --2- 172.21.15.27:0/1451083157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0700fa9d0 0x7fe0700fadf0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fe06000b5f0 tx=0x7fe060006ee0).stop 2023-11-30T21:43:30.674 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.669+0000 7fe06f59e700 1 -- 172.21.15.27:0/1451083157 shutdown_connections 2023-11-30T21:43:30.676 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.669+0000 7fe06f59e700 1 --2- 172.21.15.27:0/1451083157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0700fa9d0 0x7fe0700fadf0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:30.676 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.669+0000 7fe06f59e700 1 -- 172.21.15.27:0/1451083157 >> 172.21.15.27:0/1451083157 conn(0x7fe0700f6210 msgr2=0x7fe0700f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:30.676 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.670+0000 7fe06f59e700 1 -- 172.21.15.27:0/1451083157 shutdown_connections 2023-11-30T21:43:30.676 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.670+0000 7fe06f59e700 1 -- 172.21.15.27:0/1451083157 wait complete. 2023-11-30T21:43:30.677 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.670+0000 7fe06f59e700 1 Processor -- start 2023-11-30T21:43:30.677 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.670+0000 7fe06f59e700 1 -- start start 2023-11-30T21:43:30.677 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.670+0000 7fe06f59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0700fa9d0 0x7fe07010a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:30.677 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.670+0000 7fe06f59e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07010acf0 con 0x7fe0700fa9d0 2023-11-30T21:43:30.677 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.670+0000 7fe06e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0700fa9d0 0x7fe07010a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:30.677 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.670+0000 7fe06e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0700fa9d0 0x7fe07010a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32948/0 (socket says 172.21.15.27:32948) 2023-11-30T21:43:30.677 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.670+0000 7fe06e59c700 1 -- 172.21.15.27:0/1189495057 learned_addr learned my addr 172.21.15.27:0/1189495057 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:30.677 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.671+0000 7fe06e59c700 1 -- 172.21.15.27:0/1189495057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06000b200 con 0x7fe0700fa9d0 2023-11-30T21:43:30.677 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.671+0000 7fe06e59c700 1 --2- 172.21.15.27:0/1189495057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0700fa9d0 0x7fe07010a800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fe06000b1d0 tx=0x7fe060006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:30.677 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.671+0000 7fe05effd700 1 -- 172.21.15.27:0/1189495057 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe06001b410 con 0x7fe0700fa9d0 2023-11-30T21:43:30.677 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.671+0000 7fe05effd700 1 -- 172.21.15.27:0/1189495057 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe06001b9e0 con 0x7fe0700fa9d0 2023-11-30T21:43:30.677 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.671+0000 7fe06f59e700 1 -- 172.21.15.27:0/1189495057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07010aef0 con 0x7fe0700fa9d0 2023-11-30T21:43:30.678 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.671+0000 7fe05effd700 1 -- 172.21.15.27:0/1189495057 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe06001a840 con 0x7fe0700fa9d0 2023-11-30T21:43:30.678 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.671+0000 7fe06f59e700 1 -- 172.21.15.27:0/1189495057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07010b300 con 0x7fe0700fa9d0 2023-11-30T21:43:30.678 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.672+0000 7fe05effd700 1 -- 172.21.15.27:0/1189495057 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fe06000f070 con 0x7fe0700fa9d0 2023-11-30T21:43:30.678 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.672+0000 7fe05effd700 1 --2- 172.21.15.27:0/1189495057 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fe05402fcf0 0x7fe0540321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:30.678 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.672+0000 7fe06f59e700 1 -- 172.21.15.27:0/1189495057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe070045ad0 con 0x7fe0700fa9d0 2023-11-30T21:43:30.678 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.672+0000 7fe05effd700 1 -- 172.21.15.27:0/1189495057 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe0600410a0 con 0x7fe0700fa9d0 2023-11-30T21:43:30.678 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.672+0000 7fe06dd9b700 1 --2- 172.21.15.27:0/1189495057 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fe05402fcf0 0x7fe0540321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:30.678 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.672+0000 7fe06dd9b700 1 --2- 172.21.15.27:0/1189495057 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fe05402fcf0 0x7fe0540321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe0640096a0 tx=0x7fe064006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:30.678 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.676+0000 7fe05effd700 1 -- 172.21.15.27:0/1189495057 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe060016070 con 0x7fe0700fa9d0 2023-11-30T21:43:30.869 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.867+0000 7fe06f59e700 1 -- 172.21.15.27:0/1189495057 --> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fe07010bc40 con 0x7fe05402fcf0 2023-11-30T21:43:30.873 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.872+0000 7fe05effd700 1 -- 172.21.15.27:0/1189495057 <== mgr.14114 v2:172.21.15.27:6800/2518541463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe07010bc40 con 0x7fe05402fcf0 2023-11-30T21:43:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.874+0000 7fe06f59e700 1 -- 172.21.15.27:0/1189495057 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fe05402fcf0 msgr2=0x7fe0540321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.874+0000 7fe06f59e700 1 --2- 172.21.15.27:0/1189495057 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fe05402fcf0 0x7fe0540321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe0640096a0 tx=0x7fe064006b40).stop 2023-11-30T21:43:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.874+0000 7fe06f59e700 1 -- 172.21.15.27:0/1189495057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0700fa9d0 msgr2=0x7fe07010a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.874+0000 7fe06f59e700 1 --2- 172.21.15.27:0/1189495057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0700fa9d0 0x7fe07010a800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fe06000b1d0 tx=0x7fe060006ee0).stop 2023-11-30T21:43:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.874+0000 7fe06f59e700 1 -- 172.21.15.27:0/1189495057 shutdown_connections 2023-11-30T21:43:30.876 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.874+0000 7fe06f59e700 1 --2- 172.21.15.27:0/1189495057 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fe05402fcf0 0x7fe0540321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.874+0000 7fe06f59e700 1 --2- 172.21.15.27:0/1189495057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0700fa9d0 0x7fe07010a800 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.874+0000 7fe06f59e700 1 -- 172.21.15.27:0/1189495057 >> 172.21.15.27:0/1189495057 conn(0x7fe0700f6210 msgr2=0x7fe070070050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.874+0000 7fe06f59e700 1 -- 172.21.15.27:0/1189495057 shutdown_connections 2023-11-30T21:43:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:30.874+0000 7fe06f59e700 1 -- 172.21.15.27:0/1189495057 wait complete. 2023-11-30T21:43:31.250 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:30 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:29.943492+0000 mgr.smithi027.brrjcl (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-30T21:43:31.250 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:30 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:29.944148+0000 mon.smithi027 (mon.0) 47 : cluster [DBG] mgrmap e6: smithi027.brrjcl(active, since 1.06318s) 2023-11-30T21:43:31.250 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:30 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:29.950690+0000 mgr.smithi027.brrjcl (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-30T21:43:31.250 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:30 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:30.873335+0000 mon.smithi027 (mon.0) 48 : audit [INF] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:31.250 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:30 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:30.877398+0000 mon.smithi027 (mon.0) 49 : audit [DBG] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-30T21:43:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-11-30T21:43:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.601+0000 7f99e5acf700 1 Processor -- start 2023-11-30T21:43:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.601+0000 7f99e5acf700 1 -- start start 2023-11-30T21:43:31.603 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.601+0000 7f99e5acf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e00fa9d0 0x7f99e00fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:31.604 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.601+0000 7f99e5acf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e00fb2e0 con 0x7f99e00fa9d0 2023-11-30T21:43:31.604 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.601+0000 7f99e4acd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e00fa9d0 0x7f99e00fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:31.604 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.601+0000 7f99e4acd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e00fa9d0 0x7f99e00fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32952/0 (socket says 172.21.15.27:32952) 2023-11-30T21:43:31.604 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.601+0000 7f99e4acd700 1 -- 172.21.15.27:0/3054388619 learned_addr learned my addr 172.21.15.27:0/3054388619 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:31.604 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.601+0000 7f99e4acd700 1 -- 172.21.15.27:0/3054388619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e00fb420 con 0x7f99e00fa9d0 2023-11-30T21:43:31.604 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.602+0000 7f99e4acd700 1 --2- 172.21.15.27:0/3054388619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e00fa9d0 0x7f99e00fadf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f99d400b5f0 tx=0x7f99d4006ee0).ready entity=mon.0 client_cookie=4e9adc60778a6c9f server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:31.604 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.602+0000 7f99df7fe700 1 -- 172.21.15.27:0/3054388619 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99d400d600 con 0x7f99e00fa9d0 2023-11-30T21:43:31.604 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.602+0000 7f99df7fe700 1 -- 172.21.15.27:0/3054388619 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f99d400dbd0 con 0x7f99e00fa9d0 2023-11-30T21:43:31.604 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.602+0000 7f99df7fe700 1 -- 172.21.15.27:0/3054388619 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99d4011c00 con 0x7f99e00fa9d0 2023-11-30T21:43:31.604 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.602+0000 7f99e5acf700 1 -- 172.21.15.27:0/3054388619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e00fa9d0 msgr2=0x7f99e00fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.602+0000 7f99e5acf700 1 --2- 172.21.15.27:0/3054388619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e00fa9d0 0x7f99e00fadf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f99d400b5f0 tx=0x7f99d4006ee0).stop 2023-11-30T21:43:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.602+0000 7f99e5acf700 1 -- 172.21.15.27:0/3054388619 shutdown_connections 2023-11-30T21:43:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.602+0000 7f99e5acf700 1 --2- 172.21.15.27:0/3054388619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e00fa9d0 0x7f99e00fadf0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.602+0000 7f99e5acf700 1 -- 172.21.15.27:0/3054388619 >> 172.21.15.27:0/3054388619 conn(0x7f99e00f6210 msgr2=0x7f99e00f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.602+0000 7f99e5acf700 1 -- 172.21.15.27:0/3054388619 shutdown_connections 2023-11-30T21:43:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.603+0000 7f99e5acf700 1 -- 172.21.15.27:0/3054388619 wait complete. 2023-11-30T21:43:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.603+0000 7f99e5acf700 1 Processor -- start 2023-11-30T21:43:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.603+0000 7f99e5acf700 1 -- start start 2023-11-30T21:43:31.606 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.604+0000 7f99e5acf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e00fa9d0 0x7f99e010d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.604+0000 7f99e5acf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e00704b0 con 0x7f99e00fa9d0 2023-11-30T21:43:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.604+0000 7f99e4acd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e00fa9d0 0x7f99e010d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.604+0000 7f99e4acd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e00fa9d0 0x7f99e010d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32960/0 (socket says 172.21.15.27:32960) 2023-11-30T21:43:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.604+0000 7f99e4acd700 1 -- 172.21.15.27:0/551214058 learned_addr learned my addr 172.21.15.27:0/551214058 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.604+0000 7f99e4acd700 1 -- 172.21.15.27:0/551214058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d400b200 con 0x7f99e00fa9d0 2023-11-30T21:43:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.604+0000 7f99e4acd700 1 --2- 172.21.15.27:0/551214058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e00fa9d0 0x7f99e010d4c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f99d400b1d0 tx=0x7f99d4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.604+0000 7f99dd7fa700 1 -- 172.21.15.27:0/551214058 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99d401b410 con 0x7f99e00fa9d0 2023-11-30T21:43:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.604+0000 7f99dd7fa700 1 -- 172.21.15.27:0/551214058 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f99d401b9e0 con 0x7f99e00fa9d0 2023-11-30T21:43:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.605+0000 7f99e5acf700 1 -- 172.21.15.27:0/551214058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e00706b0 con 0x7f99e00fa9d0 2023-11-30T21:43:31.607 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.605+0000 7f99dd7fa700 1 -- 172.21.15.27:0/551214058 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99d401a840 con 0x7f99e00fa9d0 2023-11-30T21:43:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.605+0000 7f99e5acf700 1 -- 172.21.15.27:0/551214058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e010db70 con 0x7f99e00fa9d0 2023-11-30T21:43:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.605+0000 7f99dd7fa700 1 -- 172.21.15.27:0/551214058 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f99d400f070 con 0x7f99e00fa9d0 2023-11-30T21:43:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.606+0000 7f99dd7fa700 1 --2- 172.21.15.27:0/551214058 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f99c402fcf0 0x7f99c40321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.606+0000 7f99dd7fa700 1 -- 172.21.15.27:0/551214058 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f99d40410b0 con 0x7f99e00fa9d0 2023-11-30T21:43:31.608 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.606+0000 7f99dffff700 1 --2- 172.21.15.27:0/551214058 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f99c402fcf0 0x7f99c40321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.606+0000 7f99e5acf700 1 -- 172.21.15.27:0/551214058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e0045ad0 con 0x7f99e00fa9d0 2023-11-30T21:43:31.609 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.606+0000 7f99dffff700 1 --2- 172.21.15.27:0/551214058 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f99c402fcf0 0x7f99c40321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f99cc0096a0 tx=0x7f99cc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:31.612 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.610+0000 7f99dd7fa700 1 -- 172.21.15.27:0/551214058 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99d4025460 con 0x7f99e00fa9d0 2023-11-30T21:43:31.775 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.774+0000 7f99e5acf700 1 -- 172.21.15.27:0/551214058 --> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f99e0057f10 con 0x7f99c402fcf0 2023-11-30T21:43:31.775 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.774+0000 7f99dd7fa700 1 -- 172.21.15.27:0/551214058 <== mgr.14114 v2:172.21.15.27:6800/2518541463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f99e0057f10 con 0x7f99c402fcf0 2023-11-30T21:43:31.775 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: value unchanged 2023-11-30T21:43:31.777 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.776+0000 7f99e5acf700 1 -- 172.21.15.27:0/551214058 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f99c402fcf0 msgr2=0x7f99c40321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:31.778 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.776+0000 7f99e5acf700 1 --2- 172.21.15.27:0/551214058 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f99c402fcf0 0x7f99c40321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f99cc0096a0 tx=0x7f99cc006b40).stop 2023-11-30T21:43:31.778 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.776+0000 7f99e5acf700 1 -- 172.21.15.27:0/551214058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e00fa9d0 msgr2=0x7f99e010d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:31.778 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.776+0000 7f99e5acf700 1 --2- 172.21.15.27:0/551214058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e00fa9d0 0x7f99e010d4c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f99d400b1d0 tx=0x7f99d4006ee0).stop 2023-11-30T21:43:31.778 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.776+0000 7f99e5acf700 1 -- 172.21.15.27:0/551214058 shutdown_connections 2023-11-30T21:43:31.778 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.776+0000 7f99e5acf700 1 --2- 172.21.15.27:0/551214058 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f99c402fcf0 0x7f99c40321b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:31.778 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.776+0000 7f99e5acf700 1 --2- 172.21.15.27:0/551214058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99e00fa9d0 0x7f99e010d4c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:31.779 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.776+0000 7f99e5acf700 1 -- 172.21.15.27:0/551214058 >> 172.21.15.27:0/551214058 conn(0x7f99e00f6210 msgr2=0x7f99e00f7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:31.779 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.776+0000 7f99e5acf700 1 -- 172.21.15.27:0/551214058 shutdown_connections 2023-11-30T21:43:31.779 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:31.776+0000 7f99e5acf700 1 -- 172.21.15.27:0/551214058 wait complete. 2023-11-30T21:43:32.158 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:30.869451+0000 mgr.smithi027.brrjcl (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-30T21:43:32.158 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:31.874382+0000 mon.smithi027 (mon.0) 50 : cluster [DBG] mgrmap e7: smithi027.brrjcl(active, since 2s) 2023-11-30T21:43:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:Generating ssh key... 2023-11-30T21:43:32.172 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-11-30T21:43:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.528+0000 7f4622041700 1 Processor -- start 2023-11-30T21:43:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.528+0000 7f4622041700 1 -- start start 2023-11-30T21:43:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.529+0000 7f4622041700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c070430 0x7f461c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.529+0000 7f4622041700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461c0fedd0 con 0x7f461c070430 2023-11-30T21:43:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.529+0000 7f461b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c070430 0x7f461c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.529+0000 7f461b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c070430 0x7f461c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60594/0 (socket says 172.21.15.27:60594) 2023-11-30T21:43:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.529+0000 7f461b7fe700 1 -- 172.21.15.27:0/3956330438 learned_addr learned my addr 172.21.15.27:0/3956330438 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.529+0000 7f461b7fe700 1 -- 172.21.15.27:0/3956330438 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461c0fef10 con 0x7f461c070430 2023-11-30T21:43:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.529+0000 7f461b7fe700 1 --2- 172.21.15.27:0/3956330438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c070430 0x7f461c0fe8e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f460c00b5f0 tx=0x7f460c006ee0).ready entity=mon.0 client_cookie=5ba2fd56e4c224be server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.529+0000 7f461a7fc700 1 -- 172.21.15.27:0/3956330438 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f460c00d600 con 0x7f461c070430 2023-11-30T21:43:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.529+0000 7f461a7fc700 1 -- 172.21.15.27:0/3956330438 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f460c00dbd0 con 0x7f461c070430 2023-11-30T21:43:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.529+0000 7f461a7fc700 1 -- 172.21.15.27:0/3956330438 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f460c011c00 con 0x7f461c070430 2023-11-30T21:43:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.530+0000 7f4622041700 1 -- 172.21.15.27:0/3956330438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c070430 msgr2=0x7f461c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.530+0000 7f4622041700 1 --2- 172.21.15.27:0/3956330438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c070430 0x7f461c0fe8e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f460c00b5f0 tx=0x7f460c006ee0).stop 2023-11-30T21:43:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.530+0000 7f4622041700 1 -- 172.21.15.27:0/3956330438 shutdown_connections 2023-11-30T21:43:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.530+0000 7f4622041700 1 --2- 172.21.15.27:0/3956330438 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c070430 0x7f461c0fe8e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.530+0000 7f4622041700 1 -- 172.21.15.27:0/3956330438 >> 172.21.15.27:0/3956330438 conn(0x7f461c0f63c0 msgr2=0x7f461c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.530+0000 7f4622041700 1 -- 172.21.15.27:0/3956330438 shutdown_connections 2023-11-30T21:43:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.530+0000 7f4622041700 1 -- 172.21.15.27:0/3956330438 wait complete. 2023-11-30T21:43:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.531+0000 7f4622041700 1 Processor -- start 2023-11-30T21:43:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.531+0000 7f4622041700 1 -- start start 2023-11-30T21:43:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.531+0000 7f4622041700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c070430 0x7f461c10aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.531+0000 7f4622041700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461c10aef0 con 0x7f461c070430 2023-11-30T21:43:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.531+0000 7f461b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c070430 0x7f461c10aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.531+0000 7f461b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c070430 0x7f461c10aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60598/0 (socket says 172.21.15.27:60598) 2023-11-30T21:43:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.531+0000 7f461b7fe700 1 -- 172.21.15.27:0/1032509162 learned_addr learned my addr 172.21.15.27:0/1032509162 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.532+0000 7f461b7fe700 1 -- 172.21.15.27:0/1032509162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c00b200 con 0x7f461c070430 2023-11-30T21:43:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.532+0000 7f461b7fe700 1 --2- 172.21.15.27:0/1032509162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c070430 0x7f461c10aa00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f460c00b190 tx=0x7f460c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.532+0000 7f4603fff700 1 -- 172.21.15.27:0/1032509162 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f460c01b410 con 0x7f461c070430 2023-11-30T21:43:32.537 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.532+0000 7f4622041700 1 -- 172.21.15.27:0/1032509162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f461c10b0f0 con 0x7f461c070430 2023-11-30T21:43:32.537 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.532+0000 7f4603fff700 1 -- 172.21.15.27:0/1032509162 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f460c01b9e0 con 0x7f461c070430 2023-11-30T21:43:32.537 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.532+0000 7f4603fff700 1 -- 172.21.15.27:0/1032509162 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f460c01a880 con 0x7f461c070430 2023-11-30T21:43:32.537 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.532+0000 7f4622041700 1 -- 172.21.15.27:0/1032509162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f461c10b500 con 0x7f461c070430 2023-11-30T21:43:32.537 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.533+0000 7f4603fff700 1 -- 172.21.15.27:0/1032509162 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f460c00f070 con 0x7f461c070430 2023-11-30T21:43:32.537 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.533+0000 7f4603fff700 1 --2- 172.21.15.27:0/1032509162 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f460802fd50 0x7f4608032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:32.537 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.533+0000 7f4603fff700 1 -- 172.21.15.27:0/1032509162 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f460c042140 con 0x7f461c070430 2023-11-30T21:43:32.537 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.533+0000 7f461affd700 1 --2- 172.21.15.27:0/1032509162 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f460802fd50 0x7f4608032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:32.537 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.533+0000 7f4622041700 1 -- 172.21.15.27:0/1032509162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461c059100 con 0x7f461c070430 2023-11-30T21:43:32.537 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.533+0000 7f461affd700 1 --2- 172.21.15.27:0/1032509162 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f460802fd50 0x7f4608032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f46100096a0 tx=0x7f4610006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:32.539 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.538+0000 7f4603fff700 1 -- 172.21.15.27:0/1032509162 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f460c010360 con 0x7f461c070430 2023-11-30T21:43:32.698 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.696+0000 7f4622041700 1 -- 172.21.15.27:0/1032509162 --> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f461c1048f0 con 0x7f460802fd50 2023-11-30T21:43:32.971 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.970+0000 7f4603fff700 1 -- 172.21.15.27:0/1032509162 <== mgr.14114 v2:172.21.15.27:6800/2518541463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f461c1048f0 con 0x7f460802fd50 2023-11-30T21:43:32.973 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.971+0000 7f4622041700 1 -- 172.21.15.27:0/1032509162 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f460802fd50 msgr2=0x7f4608032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:32.973 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.971+0000 7f4622041700 1 --2- 172.21.15.27:0/1032509162 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f460802fd50 0x7f4608032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f46100096a0 tx=0x7f4610006b40).stop 2023-11-30T21:43:32.973 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.971+0000 7f4622041700 1 -- 172.21.15.27:0/1032509162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c070430 msgr2=0x7f461c10aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:32.973 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.971+0000 7f4622041700 1 --2- 172.21.15.27:0/1032509162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c070430 0x7f461c10aa00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f460c00b190 tx=0x7f460c006ee0).stop 2023-11-30T21:43:32.974 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.971+0000 7f4622041700 1 -- 172.21.15.27:0/1032509162 shutdown_connections 2023-11-30T21:43:32.974 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.971+0000 7f4622041700 1 --2- 172.21.15.27:0/1032509162 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f460802fd50 0x7f4608032210 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:32.974 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.971+0000 7f4622041700 1 --2- 172.21.15.27:0/1032509162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c070430 0x7f461c10aa00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:32.974 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.971+0000 7f4622041700 1 -- 172.21.15.27:0/1032509162 >> 172.21.15.27:0/1032509162 conn(0x7f461c0f63c0 msgr2=0x7f461c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:32.974 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.972+0000 7f4622041700 1 -- 172.21.15.27:0/1032509162 shutdown_connections 2023-11-30T21:43:32.974 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:32.972+0000 7f4622041700 1 -- 172.21.15.27:0/1032509162 wait complete. 2023-11-30T21:43:33.261 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:32 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:43:32.698528+0000 mgr.smithi027.brrjcl (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2023-11-30T21:43:33.314 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-11-30T21:43:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.984+0000 7fc44eca3700 1 Processor -- start 2023-11-30T21:43:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.984+0000 7fc44eca3700 1 -- start start 2023-11-30T21:43:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.984+0000 7fc44eca3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc448070430 0x7fc4480fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.985+0000 7fc44eca3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4480fedd0 con 0x7fc448070430 2023-11-30T21:43:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.985+0000 7fc44ca3f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc448070430 0x7fc4480fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.985+0000 7fc44ca3f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc448070430 0x7fc4480fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60614/0 (socket says 172.21.15.27:60614) 2023-11-30T21:43:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.985+0000 7fc44ca3f700 1 -- 172.21.15.27:0/770547366 learned_addr learned my addr 172.21.15.27:0/770547366 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.985+0000 7fc44ca3f700 1 -- 172.21.15.27:0/770547366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4480fef10 con 0x7fc448070430 2023-11-30T21:43:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.985+0000 7fc44ca3f700 1 --2- 172.21.15.27:0/770547366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc448070430 0x7fc4480fe8e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fc43c00b5f0 tx=0x7fc43c006ee0).ready entity=mon.0 client_cookie=611a26e173760be2 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.985+0000 7fc4477fe700 1 -- 172.21.15.27:0/770547366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc43c00d600 con 0x7fc448070430 2023-11-30T21:43:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.985+0000 7fc4477fe700 1 -- 172.21.15.27:0/770547366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc43c00dbd0 con 0x7fc448070430 2023-11-30T21:43:34.189 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.985+0000 7fc4477fe700 1 -- 172.21.15.27:0/770547366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc43c011c00 con 0x7fc448070430 2023-11-30T21:43:34.189 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.986+0000 7fc44eca3700 1 -- 172.21.15.27:0/770547366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc448070430 msgr2=0x7fc4480fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:34.189 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.986+0000 7fc44eca3700 1 --2- 172.21.15.27:0/770547366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc448070430 0x7fc4480fe8e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fc43c00b5f0 tx=0x7fc43c006ee0).stop 2023-11-30T21:43:34.189 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.986+0000 7fc44eca3700 1 -- 172.21.15.27:0/770547366 shutdown_connections 2023-11-30T21:43:34.189 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.986+0000 7fc44eca3700 1 --2- 172.21.15.27:0/770547366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc448070430 0x7fc4480fe8e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:34.189 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.986+0000 7fc44eca3700 1 -- 172.21.15.27:0/770547366 >> 172.21.15.27:0/770547366 conn(0x7fc4480f63c0 msgr2=0x7fc4480f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:34.189 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.986+0000 7fc44eca3700 1 -- 172.21.15.27:0/770547366 shutdown_connections 2023-11-30T21:43:34.189 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.986+0000 7fc44eca3700 1 -- 172.21.15.27:0/770547366 wait complete. 2023-11-30T21:43:34.189 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.986+0000 7fc44eca3700 1 Processor -- start 2023-11-30T21:43:34.189 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.987+0000 7fc44eca3700 1 -- start start 2023-11-30T21:43:34.189 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.987+0000 7fc44eca3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc448070430 0x7fc4480fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.987+0000 7fc44eca3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4480fa7e0 con 0x7fc448070430 2023-11-30T21:43:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.987+0000 7fc44ca3f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc448070430 0x7fc4480fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.987+0000 7fc44ca3f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc448070430 0x7fc4480fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60616/0 (socket says 172.21.15.27:60616) 2023-11-30T21:43:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.987+0000 7fc44ca3f700 1 -- 172.21.15.27:0/519161802 learned_addr learned my addr 172.21.15.27:0/519161802 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.987+0000 7fc44ca3f700 1 -- 172.21.15.27:0/519161802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c00b200 con 0x7fc448070430 2023-11-30T21:43:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.987+0000 7fc44ca3f700 1 --2- 172.21.15.27:0/519161802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc448070430 0x7fc4480fc360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc43c00b190 tx=0x7fc43c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.987+0000 7fc4457fa700 1 -- 172.21.15.27:0/519161802 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc43c01b410 con 0x7fc448070430 2023-11-30T21:43:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.987+0000 7fc4457fa700 1 -- 172.21.15.27:0/519161802 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc43c01b9e0 con 0x7fc448070430 2023-11-30T21:43:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.987+0000 7fc44eca3700 1 -- 172.21.15.27:0/519161802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4480fa980 con 0x7fc448070430 2023-11-30T21:43:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.987+0000 7fc4457fa700 1 -- 172.21.15.27:0/519161802 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc43c01a880 con 0x7fc448070430 2023-11-30T21:43:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.988+0000 7fc44eca3700 1 -- 172.21.15.27:0/519161802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4480fad90 con 0x7fc448070430 2023-11-30T21:43:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.988+0000 7fc4457fa700 1 -- 172.21.15.27:0/519161802 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc43c00f070 con 0x7fc448070430 2023-11-30T21:43:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.988+0000 7fc4457fa700 1 --2- 172.21.15.27:0/519161802 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fc42c02fdb0 0x7fc42c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.988+0000 7fc4457fa700 1 -- 172.21.15.27:0/519161802 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc43c041210 con 0x7fc448070430 2023-11-30T21:43:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.988+0000 7fc44eca3700 1 -- 172.21.15.27:0/519161802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc448045ad0 con 0x7fc448070430 2023-11-30T21:43:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.989+0000 7fc447fff700 1 --2- 172.21.15.27:0/519161802 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fc42c02fdb0 0x7fc42c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.989+0000 7fc447fff700 1 --2- 172.21.15.27:0/519161802 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fc42c02fdb0 0x7fc42c032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc4340096a0 tx=0x7fc434006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:33.993+0000 7fc4457fa700 1 -- 172.21.15.27:0/519161802 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc43c016070 con 0x7fc448070430 2023-11-30T21:43:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:34.154+0000 7fc44eca3700 1 -- 172.21.15.27:0/519161802 --> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fc448104890 con 0x7fc42c02fdb0 2023-11-30T21:43:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:34.155+0000 7fc4457fa700 1 -- 172.21.15.27:0/519161802 <== mgr.14114 v2:172.21.15.27:6800/2518541463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fc448104890 con 0x7fc42c02fdb0 2023-11-30T21:43:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:34.156+0000 7fc44eca3700 1 -- 172.21.15.27:0/519161802 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fc42c02fdb0 msgr2=0x7fc42c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:34.156+0000 7fc44eca3700 1 --2- 172.21.15.27:0/519161802 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fc42c02fdb0 0x7fc42c032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc4340096a0 tx=0x7fc434006b40).stop 2023-11-30T21:43:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:34.156+0000 7fc44eca3700 1 -- 172.21.15.27:0/519161802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc448070430 msgr2=0x7fc4480fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:34.156+0000 7fc44eca3700 1 --2- 172.21.15.27:0/519161802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc448070430 0x7fc4480fc360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc43c00b190 tx=0x7fc43c006ee0).stop 2023-11-30T21:43:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:34.157+0000 7fc44eca3700 1 -- 172.21.15.27:0/519161802 shutdown_connections 2023-11-30T21:43:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:34.157+0000 7fc44eca3700 1 --2- 172.21.15.27:0/519161802 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fc42c02fdb0 0x7fc42c032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:34.157+0000 7fc44eca3700 1 --2- 172.21.15.27:0/519161802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc448070430 0x7fc4480fc360 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:34.157+0000 7fc44eca3700 1 -- 172.21.15.27:0/519161802 >> 172.21.15.27:0/519161802 conn(0x7fc4480f63c0 msgr2=0x7fc4480f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:34.157+0000 7fc44eca3700 1 -- 172.21.15.27:0/519161802 shutdown_connections 2023-11-30T21:43:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:34.157+0000 7fc44eca3700 1 -- 172.21.15.27:0/519161802 wait complete. 2023-11-30T21:43:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7p9dzOsiDJ7AAcUjutfye6E+r288LGQ7xmovbAfEysABv2cizir0UhGNWQ5M0xpAsJd+mGozG4KuyCuYNd/LcJ4pNZ0GA7B3MlDUDPAaw06K5vmthksG5JHhRQ0LnAsH/BDmggUAxDuKWxDk5dpzoFxZVoPalt0LUROqrcqUjPG01gxPCka7ZRCN6LzK8w4P0v2x65BHlsNWe9XeSyc23vEQ7k0tDL+URNMvzAzjr7SmA7nCUga73LxuuPAw1okdMjDgtDJp27Y0RQE9mJcxIeupCFlKKpCjvSaOc23rWWjxsZkz9t6Xpq/axB4OTuzY1aAEVjBRbWWOe5QAZwlADrIHFypvSMRLTxL6XTm81LyF9YTTkK+WKTinX3qb/hGySBPMv7D9zpnINlBdLtaKn1wlI3JusJIWPEwcevnBR7OFBOw+YWSpBpyWolyvrkAH9BePgFcXEQvEnkxSsrOWBJOfbd0QtYYZvsmm+f8Vg3+VfV4S4lK0sOsMO2XqHxP8= ceph-6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:43:34.260 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:33 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:32.968443+0000 mon.smithi027 (mon.0) 51 : audit [INF] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:34.260 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:33 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:32.970128+0000 mon.smithi027 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:35.420 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:35 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:34.156107+0000 mgr.smithi027.brrjcl (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-30T21:43:35.692 INFO:teuthology.orchestra.run.smithi027.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-30T21:43:35.692 INFO:teuthology.orchestra.run.smithi027.stderr:Adding key to root@localhost authorized_keys... 2023-11-30T21:43:35.693 INFO:teuthology.orchestra.run.smithi027.stderr:Adding host smithi027... 2023-11-30T21:43:35.693 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi027 172.21.15.27 2023-11-30T21:43:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.337+0000 7ff804ff4700 1 Processor -- start 2023-11-30T21:43:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.337+0000 7ff804ff4700 1 -- start start 2023-11-30T21:43:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.337+0000 7ff804ff4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff800070430 0x7ff8000fe900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.337+0000 7ff804ff4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8000fedf0 con 0x7ff800070430 2023-11-30T21:43:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.337+0000 7ff7fe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff800070430 0x7ff8000fe900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.337+0000 7ff7fe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff800070430 0x7ff8000fe900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60630/0 (socket says 172.21.15.27:60630) 2023-11-30T21:43:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.337+0000 7ff7fe59c700 1 -- 172.21.15.27:0/2119806644 learned_addr learned my addr 172.21.15.27:0/2119806644 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.338+0000 7ff7fe59c700 1 -- 172.21.15.27:0/2119806644 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8000fef30 con 0x7ff800070430 2023-11-30T21:43:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.338+0000 7ff7fe59c700 1 --2- 172.21.15.27:0/2119806644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff800070430 0x7ff8000fe900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7ff7f000b5f0 tx=0x7ff7f0006ee0).ready entity=mon.0 client_cookie=2367004770eb1122 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.338+0000 7ff7fd59a700 1 -- 172.21.15.27:0/2119806644 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f000d600 con 0x7ff800070430 2023-11-30T21:43:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.338+0000 7ff7fd59a700 1 -- 172.21.15.27:0/2119806644 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff7f000dbd0 con 0x7ff800070430 2023-11-30T21:43:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.338+0000 7ff7fd59a700 1 -- 172.21.15.27:0/2119806644 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f0011c00 con 0x7ff800070430 2023-11-30T21:43:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.338+0000 7ff804ff4700 1 -- 172.21.15.27:0/2119806644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff800070430 msgr2=0x7ff8000fe900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.338+0000 7ff804ff4700 1 --2- 172.21.15.27:0/2119806644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff800070430 0x7ff8000fe900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7ff7f000b5f0 tx=0x7ff7f0006ee0).stop 2023-11-30T21:43:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.338+0000 7ff804ff4700 1 -- 172.21.15.27:0/2119806644 shutdown_connections 2023-11-30T21:43:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.339+0000 7ff804ff4700 1 --2- 172.21.15.27:0/2119806644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff800070430 0x7ff8000fe900 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.339+0000 7ff804ff4700 1 -- 172.21.15.27:0/2119806644 >> 172.21.15.27:0/2119806644 conn(0x7ff8000f63c0 msgr2=0x7ff8000f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:37.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.339+0000 7ff804ff4700 1 -- 172.21.15.27:0/2119806644 shutdown_connections 2023-11-30T21:43:37.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.339+0000 7ff804ff4700 1 -- 172.21.15.27:0/2119806644 wait complete. 2023-11-30T21:43:37.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.339+0000 7ff804ff4700 1 Processor -- start 2023-11-30T21:43:37.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.339+0000 7ff804ff4700 1 -- start start 2023-11-30T21:43:37.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.339+0000 7ff804ff4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff800070430 0x7ff80010aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:37.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.339+0000 7ff804ff4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80010af40 con 0x7ff800070430 2023-11-30T21:43:37.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.339+0000 7ff7fe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff800070430 0x7ff80010aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:37.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.339+0000 7ff7fe59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff800070430 0x7ff80010aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60636/0 (socket says 172.21.15.27:60636) 2023-11-30T21:43:37.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.339+0000 7ff7fe59c700 1 -- 172.21.15.27:0/524349607 learned_addr learned my addr 172.21.15.27:0/524349607 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:37.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.340+0000 7ff7fe59c700 1 -- 172.21.15.27:0/524349607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f000b200 con 0x7ff800070430 2023-11-30T21:43:37.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.340+0000 7ff7fe59c700 1 --2- 172.21.15.27:0/524349607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff800070430 0x7ff80010aa50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ff7f000e040 tx=0x7ff7f0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:37.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.340+0000 7ff7eeffd700 1 -- 172.21.15.27:0/524349607 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f000d9f0 con 0x7ff800070430 2023-11-30T21:43:37.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.340+0000 7ff7eeffd700 1 -- 172.21.15.27:0/524349607 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff7f001a9f0 con 0x7ff800070430 2023-11-30T21:43:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.340+0000 7ff804ff4700 1 -- 172.21.15.27:0/524349607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80010b140 con 0x7ff800070430 2023-11-30T21:43:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.340+0000 7ff7eeffd700 1 -- 172.21.15.27:0/524349607 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7f001b8c0 con 0x7ff800070430 2023-11-30T21:43:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.340+0000 7ff804ff4700 1 -- 172.21.15.27:0/524349607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80010b550 con 0x7ff800070430 2023-11-30T21:43:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.341+0000 7ff7eeffd700 1 -- 172.21.15.27:0/524349607 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff7f000f070 con 0x7ff800070430 2023-11-30T21:43:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.341+0000 7ff7eeffd700 1 --2- 172.21.15.27:0/524349607 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7ff7e402fd50 0x7ff7e4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.341+0000 7ff7eeffd700 1 -- 172.21.15.27:0/524349607 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff7f00413b0 con 0x7ff800070430 2023-11-30T21:43:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.341+0000 7ff804ff4700 1 -- 172.21.15.27:0/524349607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff800045ad0 con 0x7ff800070430 2023-11-30T21:43:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.341+0000 7ff7fdd9b700 1 --2- 172.21.15.27:0/524349607 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7ff7e402fd50 0x7ff7e4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.341+0000 7ff7fdd9b700 1 --2- 172.21.15.27:0/524349607 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7ff7e402fd50 0x7ff7e4032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7ff7f40096a0 tx=0x7ff7f4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.345+0000 7ff7eeffd700 1 -- 172.21.15.27:0/524349607 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff7f00224f0 con 0x7ff800070430 2023-11-30T21:43:37.529 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:37.528+0000 7ff804ff4700 1 -- 172.21.15.27:0/524349607 --> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}) v1 -- 0x7ff80010be90 con 0x7ff7e402fd50 2023-11-30T21:43:38.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:37 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:37.529952+0000 mgr.smithi027.brrjcl (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}]: dispatch 2023-11-30T21:43:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: Added host 'smithi027' 2023-11-30T21:43:38.222 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:38.220+0000 7ff7eeffd700 1 -- 172.21.15.27:0/524349607 <== mgr.14114 v2:172.21.15.27:6800/2518541463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7ff80010be90 con 0x7ff7e402fd50 2023-11-30T21:43:38.224 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:38.223+0000 7ff804ff4700 1 -- 172.21.15.27:0/524349607 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7ff7e402fd50 msgr2=0x7ff7e4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:38.224 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:38.223+0000 7ff804ff4700 1 --2- 172.21.15.27:0/524349607 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7ff7e402fd50 0x7ff7e4032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7ff7f40096a0 tx=0x7ff7f4006b40).stop 2023-11-30T21:43:38.224 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:38.223+0000 7ff804ff4700 1 -- 172.21.15.27:0/524349607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff800070430 msgr2=0x7ff80010aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:38.224 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:38.223+0000 7ff804ff4700 1 --2- 172.21.15.27:0/524349607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff800070430 0x7ff80010aa50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ff7f000e040 tx=0x7ff7f0006ee0).stop 2023-11-30T21:43:38.224 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:38.223+0000 7ff804ff4700 1 -- 172.21.15.27:0/524349607 shutdown_connections 2023-11-30T21:43:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:38.223+0000 7ff804ff4700 1 --2- 172.21.15.27:0/524349607 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7ff7e402fd50 0x7ff7e4032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:38.223+0000 7ff804ff4700 1 --2- 172.21.15.27:0/524349607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff800070430 0x7ff80010aa50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:38.223+0000 7ff804ff4700 1 -- 172.21.15.27:0/524349607 >> 172.21.15.27:0/524349607 conn(0x7ff8000f63c0 msgr2=0x7ff8000f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:38.223+0000 7ff804ff4700 1 -- 172.21.15.27:0/524349607 shutdown_connections 2023-11-30T21:43:38.225 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:38.223+0000 7ff804ff4700 1 -- 172.21.15.27:0/524349607 wait complete. 2023-11-30T21:43:39.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:38.220670+0000 mon.smithi027 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:39.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:43:38.220997+0000 mgr.smithi027.brrjcl (mgr.14114) 9 : cephadm [INF] Added host smithi027 2023-11-30T21:43:39.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:38.221217+0000 mon.smithi027 (mon.0) 54 : audit [DBG] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-30T21:43:40.388 INFO:teuthology.orchestra.run.smithi027.stderr:Deploying mon service with default placement... 2023-11-30T21:43:40.388 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-11-30T21:43:41.065 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.061+0000 7f02ef59e700 1 Processor -- start 2023-11-30T21:43:41.065 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.061+0000 7f02ef59e700 1 -- start start 2023-11-30T21:43:41.065 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.061+0000 7f02ef59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f0070430 0x7f02f00fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:41.065 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.061+0000 7f02ef59e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f00fec20 con 0x7f02f0070430 2023-11-30T21:43:41.065 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.061+0000 7f02ee59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f0070430 0x7f02f00fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.061+0000 7f02ee59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f0070430 0x7f02f00fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60648/0 (socket says 172.21.15.27:60648) 2023-11-30T21:43:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.061+0000 7f02ee59c700 1 -- 172.21.15.27:0/4100158224 learned_addr learned my addr 172.21.15.27:0/4100158224 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.062+0000 7f02ee59c700 1 -- 172.21.15.27:0/4100158224 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f00fed60 con 0x7f02f0070430 2023-11-30T21:43:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.062+0000 7f02ee59c700 1 --2- 172.21.15.27:0/4100158224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f0070430 0x7f02f00fe730 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f02e000b5f0 tx=0x7f02e0006ee0).ready entity=mon.0 client_cookie=eb95be81b9c99d1 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.062+0000 7f02ed59a700 1 -- 172.21.15.27:0/4100158224 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02e000d600 con 0x7f02f0070430 2023-11-30T21:43:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.062+0000 7f02ed59a700 1 -- 172.21.15.27:0/4100158224 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f02e000dbd0 con 0x7f02f0070430 2023-11-30T21:43:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.062+0000 7f02ed59a700 1 -- 172.21.15.27:0/4100158224 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02e0011c00 con 0x7f02f0070430 2023-11-30T21:43:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.062+0000 7f02ef59e700 1 -- 172.21.15.27:0/4100158224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f0070430 msgr2=0x7f02f00fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.062+0000 7f02ef59e700 1 --2- 172.21.15.27:0/4100158224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f0070430 0x7f02f00fe730 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f02e000b5f0 tx=0x7f02e0006ee0).stop 2023-11-30T21:43:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.063+0000 7f02ef59e700 1 -- 172.21.15.27:0/4100158224 shutdown_connections 2023-11-30T21:43:41.066 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.063+0000 7f02ef59e700 1 --2- 172.21.15.27:0/4100158224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f0070430 0x7f02f00fe730 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.063+0000 7f02ef59e700 1 -- 172.21.15.27:0/4100158224 >> 172.21.15.27:0/4100158224 conn(0x7f02f00f6210 msgr2=0x7f02f00f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.063+0000 7f02ef59e700 1 -- 172.21.15.27:0/4100158224 shutdown_connections 2023-11-30T21:43:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.063+0000 7f02ef59e700 1 -- 172.21.15.27:0/4100158224 wait complete. 2023-11-30T21:43:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.063+0000 7f02ef59e700 1 Processor -- start 2023-11-30T21:43:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.063+0000 7f02ef59e700 1 -- start start 2023-11-30T21:43:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.064+0000 7f02ef59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f0070430 0x7f02f010a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.064+0000 7f02ef59e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f010ad90 con 0x7f02f0070430 2023-11-30T21:43:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.064+0000 7f02ee59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f0070430 0x7f02f010a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.064+0000 7f02ee59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f0070430 0x7f02f010a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60660/0 (socket says 172.21.15.27:60660) 2023-11-30T21:43:41.067 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.064+0000 7f02ee59c700 1 -- 172.21.15.27:0/480811822 learned_addr learned my addr 172.21.15.27:0/480811822 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.064+0000 7f02ee59c700 1 -- 172.21.15.27:0/480811822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e000b200 con 0x7f02f0070430 2023-11-30T21:43:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.064+0000 7f02ee59c700 1 --2- 172.21.15.27:0/480811822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f0070430 0x7f02f010a8a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f02e000e040 tx=0x7f02e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.064+0000 7f02deffd700 1 -- 172.21.15.27:0/480811822 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02e000d9f0 con 0x7f02f0070430 2023-11-30T21:43:41.068 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.064+0000 7f02deffd700 1 -- 172.21.15.27:0/480811822 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f02e001a9f0 con 0x7f02f0070430 2023-11-30T21:43:41.069 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.064+0000 7f02ef59e700 1 -- 172.21.15.27:0/480811822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f010af90 con 0x7f02f0070430 2023-11-30T21:43:41.069 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.064+0000 7f02deffd700 1 -- 172.21.15.27:0/480811822 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02e001b8c0 con 0x7f02f0070430 2023-11-30T21:43:41.069 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.064+0000 7f02ef59e700 1 -- 172.21.15.27:0/480811822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f010b3a0 con 0x7f02f0070430 2023-11-30T21:43:41.069 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.065+0000 7f02deffd700 1 -- 172.21.15.27:0/480811822 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f02e000f070 con 0x7f02f0070430 2023-11-30T21:43:41.069 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.065+0000 7f02ef59e700 1 -- 172.21.15.27:0/480811822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f0045ad0 con 0x7f02f0070430 2023-11-30T21:43:41.069 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.065+0000 7f02deffd700 1 --2- 172.21.15.27:0/480811822 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f02d402fdb0 0x7f02d4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:41.069 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.066+0000 7f02deffd700 1 -- 172.21.15.27:0/480811822 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f02e0041380 con 0x7f02f0070430 2023-11-30T21:43:41.069 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.066+0000 7f02edd9b700 1 --2- 172.21.15.27:0/480811822 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f02d402fdb0 0x7f02d4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:41.069 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.066+0000 7f02edd9b700 1 --2- 172.21.15.27:0/480811822 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f02d402fdb0 0x7f02d4032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f02e40096a0 tx=0x7f02e4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:41.071 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.070+0000 7f02deffd700 1 -- 172.21.15.27:0/480811822 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02e0019d20 con 0x7f02f0070430 2023-11-30T21:43:41.257 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.255+0000 7f02ef59e700 1 -- 172.21.15.27:0/480811822 --> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f02f010bc80 con 0x7f02d402fdb0 2023-11-30T21:43:41.261 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.260+0000 7f02deffd700 1 -- 172.21.15.27:0/480811822 <== mgr.14114 v2:172.21.15.27:6800/2518541463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f02f010bc80 con 0x7f02d402fdb0 2023-11-30T21:43:41.261 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: Scheduled mon update... 2023-11-30T21:43:41.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.261+0000 7f02ef59e700 1 -- 172.21.15.27:0/480811822 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f02d402fdb0 msgr2=0x7f02d4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:41.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.261+0000 7f02ef59e700 1 --2- 172.21.15.27:0/480811822 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f02d402fdb0 0x7f02d4032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f02e40096a0 tx=0x7f02e4006b40).stop 2023-11-30T21:43:41.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.261+0000 7f02ef59e700 1 -- 172.21.15.27:0/480811822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f0070430 msgr2=0x7f02f010a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:41.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.261+0000 7f02ef59e700 1 --2- 172.21.15.27:0/480811822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f0070430 0x7f02f010a8a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f02e000e040 tx=0x7f02e0006ee0).stop 2023-11-30T21:43:41.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.261+0000 7f02ef59e700 1 -- 172.21.15.27:0/480811822 shutdown_connections 2023-11-30T21:43:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.261+0000 7f02ef59e700 1 --2- 172.21.15.27:0/480811822 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f02d402fdb0 0x7f02d4032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.261+0000 7f02ef59e700 1 --2- 172.21.15.27:0/480811822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f0070430 0x7f02f010a8a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.261+0000 7f02ef59e700 1 -- 172.21.15.27:0/480811822 >> 172.21.15.27:0/480811822 conn(0x7f02f00f6210 msgr2=0x7f02f00f7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.261+0000 7f02ef59e700 1 -- 172.21.15.27:0/480811822 shutdown_connections 2023-11-30T21:43:41.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:41.261+0000 7f02ef59e700 1 -- 172.21.15.27:0/480811822 wait complete. 2023-11-30T21:43:41.763 INFO:teuthology.orchestra.run.smithi027.stderr:Deploying mgr service with default placement... 2023-11-30T21:43:41.763 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-11-30T21:43:42.122 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.119+0000 7ff54c5ac700 1 Processor -- start 2023-11-30T21:43:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.119+0000 7ff54c5ac700 1 -- start start 2023-11-30T21:43:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.120+0000 7ff54c5ac700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5440fab80 0x7ff5440fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.120+0000 7ff54c5ac700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5440fb490 con 0x7ff5440fab80 2023-11-30T21:43:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.120+0000 7ff54a348700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5440fab80 0x7ff5440fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.120+0000 7ff54a348700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5440fab80 0x7ff5440fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60672/0 (socket says 172.21.15.27:60672) 2023-11-30T21:43:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.120+0000 7ff54a348700 1 -- 172.21.15.27:0/2532196841 learned_addr learned my addr 172.21.15.27:0/2532196841 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.120+0000 7ff54a348700 1 -- 172.21.15.27:0/2532196841 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5440fb5d0 con 0x7ff5440fab80 2023-11-30T21:43:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.120+0000 7ff54a348700 1 --2- 172.21.15.27:0/2532196841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5440fab80 0x7ff5440fafa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ff54000b870 tx=0x7ff540006f70).ready entity=mon.0 client_cookie=99385fd123564fc5 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.120+0000 7ff549346700 1 -- 172.21.15.27:0/2532196841 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff54000da20 con 0x7ff5440fab80 2023-11-30T21:43:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.120+0000 7ff549346700 1 -- 172.21.15.27:0/2532196841 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff540010470 con 0x7ff5440fab80 2023-11-30T21:43:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.121+0000 7ff549346700 1 -- 172.21.15.27:0/2532196841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff54000f5b0 con 0x7ff5440fab80 2023-11-30T21:43:42.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.121+0000 7ff54c5ac700 1 -- 172.21.15.27:0/2532196841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5440fab80 msgr2=0x7ff5440fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:42.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.121+0000 7ff54c5ac700 1 --2- 172.21.15.27:0/2532196841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5440fab80 0x7ff5440fafa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7ff54000b870 tx=0x7ff540006f70).stop 2023-11-30T21:43:42.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.121+0000 7ff54c5ac700 1 -- 172.21.15.27:0/2532196841 shutdown_connections 2023-11-30T21:43:42.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.121+0000 7ff54c5ac700 1 --2- 172.21.15.27:0/2532196841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5440fab80 0x7ff5440fafa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:42.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.121+0000 7ff54c5ac700 1 -- 172.21.15.27:0/2532196841 >> 172.21.15.27:0/2532196841 conn(0x7ff5440f63c0 msgr2=0x7ff5440f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:42.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.121+0000 7ff54c5ac700 1 -- 172.21.15.27:0/2532196841 shutdown_connections 2023-11-30T21:43:42.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.121+0000 7ff54c5ac700 1 -- 172.21.15.27:0/2532196841 wait complete. 2023-11-30T21:43:42.125 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.122+0000 7ff54c5ac700 1 Processor -- start 2023-11-30T21:43:42.125 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.122+0000 7ff54c5ac700 1 -- start start 2023-11-30T21:43:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.122+0000 7ff54c5ac700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5440fab80 0x7ff544108770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.122+0000 7ff54c5ac700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff544108c60 con 0x7ff5440fab80 2023-11-30T21:43:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.122+0000 7ff54a348700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5440fab80 0x7ff544108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.123+0000 7ff54a348700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5440fab80 0x7ff544108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60688/0 (socket says 172.21.15.27:60688) 2023-11-30T21:43:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.123+0000 7ff54a348700 1 -- 172.21.15.27:0/3469932148 learned_addr learned my addr 172.21.15.27:0/3469932148 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.123+0000 7ff54a348700 1 -- 172.21.15.27:0/3469932148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54000b480 con 0x7ff5440fab80 2023-11-30T21:43:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.123+0000 7ff54a348700 1 --2- 172.21.15.27:0/3469932148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5440fab80 0x7ff544108770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ff540006800 tx=0x7ff540006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.123+0000 7ff532ffd700 1 -- 172.21.15.27:0/3469932148 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff54000f820 con 0x7ff5440fab80 2023-11-30T21:43:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.123+0000 7ff532ffd700 1 -- 172.21.15.27:0/3469932148 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff54000fdf0 con 0x7ff5440fab80 2023-11-30T21:43:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.123+0000 7ff54c5ac700 1 -- 172.21.15.27:0/3469932148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff544108e60 con 0x7ff5440fab80 2023-11-30T21:43:42.126 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.123+0000 7ff532ffd700 1 -- 172.21.15.27:0/3469932148 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff540019d50 con 0x7ff5440fab80 2023-11-30T21:43:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.123+0000 7ff54c5ac700 1 -- 172.21.15.27:0/3469932148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff544109270 con 0x7ff5440fab80 2023-11-30T21:43:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.124+0000 7ff532ffd700 1 -- 172.21.15.27:0/3469932148 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff540016030 con 0x7ff5440fab80 2023-11-30T21:43:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.124+0000 7ff532ffd700 1 --2- 172.21.15.27:0/3469932148 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7ff52c02fdb0 0x7ff52c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.124+0000 7ff532ffd700 1 -- 172.21.15.27:0/3469932148 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff540043f00 con 0x7ff5440fab80 2023-11-30T21:43:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.124+0000 7ff549b47700 1 --2- 172.21.15.27:0/3469932148 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7ff52c02fdb0 0x7ff52c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.124+0000 7ff54c5ac700 1 -- 172.21.15.27:0/3469932148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff544059100 con 0x7ff5440fab80 2023-11-30T21:43:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.125+0000 7ff549b47700 1 --2- 172.21.15.27:0/3469932148 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7ff52c02fdb0 0x7ff52c032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7ff5340096a0 tx=0x7ff534006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.129+0000 7ff532ffd700 1 -- 172.21.15.27:0/3469932148 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff540014080 con 0x7ff5440fab80 2023-11-30T21:43:42.318 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.317+0000 7ff54c5ac700 1 -- 172.21.15.27:0/3469932148 --> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7ff544102640 con 0x7ff52c02fdb0 2023-11-30T21:43:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.321+0000 7ff532ffd700 1 -- 172.21.15.27:0/3469932148 <== mgr.14114 v2:172.21.15.27:6800/2518541463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff544102640 con 0x7ff52c02fdb0 2023-11-30T21:43:42.322 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-11-30T21:43:42.325 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.323+0000 7ff54c5ac700 1 -- 172.21.15.27:0/3469932148 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7ff52c02fdb0 msgr2=0x7ff52c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:42.325 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.323+0000 7ff54c5ac700 1 --2- 172.21.15.27:0/3469932148 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7ff52c02fdb0 0x7ff52c032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7ff5340096a0 tx=0x7ff534006b40).stop 2023-11-30T21:43:42.325 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.323+0000 7ff54c5ac700 1 -- 172.21.15.27:0/3469932148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5440fab80 msgr2=0x7ff544108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:42.325 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.323+0000 7ff54c5ac700 1 --2- 172.21.15.27:0/3469932148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5440fab80 0x7ff544108770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ff540006800 tx=0x7ff540006f70).stop 2023-11-30T21:43:42.325 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.323+0000 7ff54c5ac700 1 -- 172.21.15.27:0/3469932148 shutdown_connections 2023-11-30T21:43:42.325 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.323+0000 7ff54c5ac700 1 --2- 172.21.15.27:0/3469932148 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7ff52c02fdb0 0x7ff52c032270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:42.326 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.323+0000 7ff54c5ac700 1 --2- 172.21.15.27:0/3469932148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5440fab80 0x7ff544108770 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:42.326 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.323+0000 7ff54c5ac700 1 -- 172.21.15.27:0/3469932148 >> 172.21.15.27:0/3469932148 conn(0x7ff5440f63c0 msgr2=0x7ff5440ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:42.326 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.323+0000 7ff54c5ac700 1 -- 172.21.15.27:0/3469932148 shutdown_connections 2023-11-30T21:43:42.326 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:42.323+0000 7ff54c5ac700 1 -- 172.21.15.27:0/3469932148 wait complete. 2023-11-30T21:43:42.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:42 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:41.257567+0000 mgr.smithi027.brrjcl (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-11-30T21:43:42.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:42 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:43:41.258292+0000 mgr.smithi027.brrjcl (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-11-30T21:43:42.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:42 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:41.260568+0000 mon.smithi027 (mon.0) 55 : audit [INF] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:42.788 INFO:teuthology.orchestra.run.smithi027.stderr:Deploying crash service with default placement... 2023-11-30T21:43:42.788 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-11-30T21:43:43.163 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.161+0000 7fbd8759e700 1 Processor -- start 2023-11-30T21:43:43.163 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.161+0000 7fbd8759e700 1 -- start start 2023-11-30T21:43:43.164 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.161+0000 7fbd8759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd880703f0 0x7fbd880fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:43.164 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.161+0000 7fbd8759e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd880fec60 con 0x7fbd880703f0 2023-11-30T21:43:43.164 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.161+0000 7fbd8659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd880703f0 0x7fbd880fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:43.164 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.161+0000 7fbd8659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd880703f0 0x7fbd880fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48414/0 (socket says 172.21.15.27:48414) 2023-11-30T21:43:43.164 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.161+0000 7fbd8659c700 1 -- 172.21.15.27:0/3801124446 learned_addr learned my addr 172.21.15.27:0/3801124446 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:43.165 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.162+0000 7fbd8659c700 1 -- 172.21.15.27:0/3801124446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd880feda0 con 0x7fbd880703f0 2023-11-30T21:43:43.165 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.162+0000 7fbd8659c700 1 --2- 172.21.15.27:0/3801124446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd880703f0 0x7fbd880fe770 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fbd7800b5f0 tx=0x7fbd78006ee0).ready entity=mon.0 client_cookie=2a33ee1fc5dda231 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:43.165 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.162+0000 7fbd8559a700 1 -- 172.21.15.27:0/3801124446 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd7800d600 con 0x7fbd880703f0 2023-11-30T21:43:43.165 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.162+0000 7fbd8559a700 1 -- 172.21.15.27:0/3801124446 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbd7800dbd0 con 0x7fbd880703f0 2023-11-30T21:43:43.165 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.162+0000 7fbd8559a700 1 -- 172.21.15.27:0/3801124446 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd78011c00 con 0x7fbd880703f0 2023-11-30T21:43:43.165 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.162+0000 7fbd8759e700 1 -- 172.21.15.27:0/3801124446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd880703f0 msgr2=0x7fbd880fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:43.168 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.162+0000 7fbd8759e700 1 --2- 172.21.15.27:0/3801124446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd880703f0 0x7fbd880fe770 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fbd7800b5f0 tx=0x7fbd78006ee0).stop 2023-11-30T21:43:43.168 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.162+0000 7fbd8759e700 1 -- 172.21.15.27:0/3801124446 shutdown_connections 2023-11-30T21:43:43.169 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.162+0000 7fbd8759e700 1 --2- 172.21.15.27:0/3801124446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd880703f0 0x7fbd880fe770 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:43.169 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.162+0000 7fbd8759e700 1 -- 172.21.15.27:0/3801124446 >> 172.21.15.27:0/3801124446 conn(0x7fbd880f6210 msgr2=0x7fbd880f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:43.169 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.163+0000 7fbd8759e700 1 -- 172.21.15.27:0/3801124446 shutdown_connections 2023-11-30T21:43:43.169 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.163+0000 7fbd8759e700 1 -- 172.21.15.27:0/3801124446 wait complete. 2023-11-30T21:43:43.170 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.163+0000 7fbd8759e700 1 Processor -- start 2023-11-30T21:43:43.170 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.163+0000 7fbd8759e700 1 -- start start 2023-11-30T21:43:43.170 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.163+0000 7fbd8759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd880703f0 0x7fbd880fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:43.170 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.163+0000 7fbd8759e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd880fa670 con 0x7fbd880703f0 2023-11-30T21:43:43.170 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.163+0000 7fbd8659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd880703f0 0x7fbd880fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:43.170 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.163+0000 7fbd8659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd880703f0 0x7fbd880fc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48422/0 (socket says 172.21.15.27:48422) 2023-11-30T21:43:43.170 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.163+0000 7fbd8659c700 1 -- 172.21.15.27:0/1456862800 learned_addr learned my addr 172.21.15.27:0/1456862800 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:43.171 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.164+0000 7fbd8659c700 1 -- 172.21.15.27:0/1456862800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7800b200 con 0x7fbd880703f0 2023-11-30T21:43:43.171 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.164+0000 7fbd8659c700 1 --2- 172.21.15.27:0/1456862800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd880703f0 0x7fbd880fc1f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fbd7800e040 tx=0x7fbd78006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:43.171 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.164+0000 7fbd76ffd700 1 -- 172.21.15.27:0/1456862800 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd7800d9f0 con 0x7fbd880703f0 2023-11-30T21:43:43.171 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.164+0000 7fbd76ffd700 1 -- 172.21.15.27:0/1456862800 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbd7801a9f0 con 0x7fbd880703f0 2023-11-30T21:43:43.172 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.164+0000 7fbd76ffd700 1 -- 172.21.15.27:0/1456862800 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd7801b8c0 con 0x7fbd880703f0 2023-11-30T21:43:43.172 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.164+0000 7fbd8759e700 1 -- 172.21.15.27:0/1456862800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd880fa810 con 0x7fbd880703f0 2023-11-30T21:43:43.172 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.164+0000 7fbd8759e700 1 -- 172.21.15.27:0/1456862800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd880fac20 con 0x7fbd880703f0 2023-11-30T21:43:43.172 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.165+0000 7fbd76ffd700 1 -- 172.21.15.27:0/1456862800 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fbd7800f070 con 0x7fbd880703f0 2023-11-30T21:43:43.172 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.165+0000 7fbd76ffd700 1 --2- 172.21.15.27:0/1456862800 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fbd6c02fd50 0x7fbd6c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:43.172 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.165+0000 7fbd76ffd700 1 -- 172.21.15.27:0/1456862800 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fbd780413d0 con 0x7fbd880703f0 2023-11-30T21:43:43.172 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.165+0000 7fbd8759e700 1 -- 172.21.15.27:0/1456862800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd88045ad0 con 0x7fbd880703f0 2023-11-30T21:43:43.172 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.165+0000 7fbd85d9b700 1 --2- 172.21.15.27:0/1456862800 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fbd6c02fd50 0x7fbd6c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:43.172 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.165+0000 7fbd85d9b700 1 --2- 172.21.15.27:0/1456862800 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fbd6c02fd50 0x7fbd6c032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fbd7c0096a0 tx=0x7fbd7c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:43.173 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.169+0000 7fbd76ffd700 1 -- 172.21.15.27:0/1456862800 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd78016070 con 0x7fbd880703f0 2023-11-30T21:43:43.356 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.355+0000 7fbd8759e700 1 -- 172.21.15.27:0/1456862800 --> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd881046b0 con 0x7fbd6c02fd50 2023-11-30T21:43:43.360 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.359+0000 7fbd76ffd700 1 -- 172.21.15.27:0/1456862800 <== mgr.14114 v2:172.21.15.27:6800/2518541463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fbd881046b0 con 0x7fbd6c02fd50 2023-11-30T21:43:43.360 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: Scheduled crash update... 2023-11-30T21:43:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.360+0000 7fbd8759e700 1 -- 172.21.15.27:0/1456862800 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fbd6c02fd50 msgr2=0x7fbd6c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.360+0000 7fbd8759e700 1 --2- 172.21.15.27:0/1456862800 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fbd6c02fd50 0x7fbd6c032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fbd7c0096a0 tx=0x7fbd7c006b40).stop 2023-11-30T21:43:43.363 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.360+0000 7fbd8759e700 1 -- 172.21.15.27:0/1456862800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd880703f0 msgr2=0x7fbd880fc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:43.363 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.360+0000 7fbd8759e700 1 --2- 172.21.15.27:0/1456862800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd880703f0 0x7fbd880fc1f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fbd7800e040 tx=0x7fbd78006ee0).stop 2023-11-30T21:43:43.363 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.360+0000 7fbd8759e700 1 -- 172.21.15.27:0/1456862800 shutdown_connections 2023-11-30T21:43:43.363 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.360+0000 7fbd8759e700 1 --2- 172.21.15.27:0/1456862800 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7fbd6c02fd50 0x7fbd6c032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:43.363 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.360+0000 7fbd8759e700 1 --2- 172.21.15.27:0/1456862800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd880703f0 0x7fbd880fc1f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:43.363 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.361+0000 7fbd8759e700 1 -- 172.21.15.27:0/1456862800 >> 172.21.15.27:0/1456862800 conn(0x7fbd880f6210 msgr2=0x7fbd880f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:43.363 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.361+0000 7fbd8759e700 1 -- 172.21.15.27:0/1456862800 shutdown_connections 2023-11-30T21:43:43.363 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:43.361+0000 7fbd8759e700 1 -- 172.21.15.27:0/1456862800 wait complete. 2023-11-30T21:43:43.678 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:42.319190+0000 mgr.smithi027.brrjcl (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-11-30T21:43:43.678 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:43:42.319690+0000 mgr.smithi027.brrjcl (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-11-30T21:43:43.678 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:42.322327+0000 mon.smithi027 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:43.871 INFO:teuthology.orchestra.run.smithi027.stderr:Enabling mgr prometheus module... 2023-11-30T21:43:43.871 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-11-30T21:43:44.214 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.211+0000 7f431948a700 1 Processor -- start 2023-11-30T21:43:44.214 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.212+0000 7f431948a700 1 -- start start 2023-11-30T21:43:44.214 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.212+0000 7f431948a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140fab90 0x7f43140fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:44.214 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.212+0000 7f431948a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43140fb4a0 con 0x7f43140fab90 2023-11-30T21:43:44.215 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.212+0000 7f4312ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140fab90 0x7f43140fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:44.215 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.212+0000 7f4312ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140fab90 0x7f43140fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48428/0 (socket says 172.21.15.27:48428) 2023-11-30T21:43:44.215 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.212+0000 7f4312ffd700 1 -- 172.21.15.27:0/758094379 learned_addr learned my addr 172.21.15.27:0/758094379 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:44.215 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.212+0000 7f4312ffd700 1 -- 172.21.15.27:0/758094379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43140fb5e0 con 0x7f43140fab90 2023-11-30T21:43:44.217 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.212+0000 7f4312ffd700 1 --2- 172.21.15.27:0/758094379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140fab90 0x7f43140fafb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f430800b870 tx=0x7f4308006f70).ready entity=mon.0 client_cookie=aa30a16994aa60c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:44.217 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.213+0000 7f4311ffb700 1 -- 172.21.15.27:0/758094379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f430800da20 con 0x7f43140fab90 2023-11-30T21:43:44.217 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.213+0000 7f4311ffb700 1 -- 172.21.15.27:0/758094379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4308010470 con 0x7f43140fab90 2023-11-30T21:43:44.217 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.213+0000 7f4311ffb700 1 -- 172.21.15.27:0/758094379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f430800f5b0 con 0x7f43140fab90 2023-11-30T21:43:44.218 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.213+0000 7f431948a700 1 -- 172.21.15.27:0/758094379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140fab90 msgr2=0x7f43140fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:44.218 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.213+0000 7f431948a700 1 --2- 172.21.15.27:0/758094379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140fab90 0x7f43140fafb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f430800b870 tx=0x7f4308006f70).stop 2023-11-30T21:43:44.218 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.213+0000 7f431948a700 1 -- 172.21.15.27:0/758094379 shutdown_connections 2023-11-30T21:43:44.218 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.213+0000 7f431948a700 1 --2- 172.21.15.27:0/758094379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140fab90 0x7f43140fafb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:44.218 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.213+0000 7f431948a700 1 -- 172.21.15.27:0/758094379 >> 172.21.15.27:0/758094379 conn(0x7f43140f63b0 msgr2=0x7f43140f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:44.218 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.213+0000 7f431948a700 1 -- 172.21.15.27:0/758094379 shutdown_connections 2023-11-30T21:43:44.218 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.213+0000 7f431948a700 1 -- 172.21.15.27:0/758094379 wait complete. 2023-11-30T21:43:44.218 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.214+0000 7f431948a700 1 Processor -- start 2023-11-30T21:43:44.218 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.214+0000 7f431948a700 1 -- start start 2023-11-30T21:43:44.218 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.214+0000 7f431948a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140fab90 0x7f431410cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:44.218 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.214+0000 7f431948a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431410d030 con 0x7f43140fab90 2023-11-30T21:43:44.219 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.214+0000 7f4312ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140fab90 0x7f431410cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:44.219 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.214+0000 7f4312ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140fab90 0x7f431410cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48438/0 (socket says 172.21.15.27:48438) 2023-11-30T21:43:44.219 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.214+0000 7f4312ffd700 1 -- 172.21.15.27:0/353172590 learned_addr learned my addr 172.21.15.27:0/353172590 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:44.219 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.214+0000 7f4312ffd700 1 -- 172.21.15.27:0/353172590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430800b480 con 0x7f43140fab90 2023-11-30T21:43:44.219 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.215+0000 7f4312ffd700 1 --2- 172.21.15.27:0/353172590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140fab90 0x7f431410cb40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f4308006a40 tx=0x7f4308006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:44.219 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.215+0000 7f42fb7fe700 1 -- 172.21.15.27:0/353172590 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f430800de00 con 0x7f43140fab90 2023-11-30T21:43:44.219 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.215+0000 7f431948a700 1 -- 172.21.15.27:0/353172590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431410d230 con 0x7f43140fab90 2023-11-30T21:43:44.219 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.215+0000 7f42fb7fe700 1 -- 172.21.15.27:0/353172590 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f430800fef0 con 0x7f43140fab90 2023-11-30T21:43:44.219 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.215+0000 7f42fb7fe700 1 -- 172.21.15.27:0/353172590 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4308019e90 con 0x7f43140fab90 2023-11-30T21:43:44.219 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.215+0000 7f431948a700 1 -- 172.21.15.27:0/353172590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431410d640 con 0x7f43140fab90 2023-11-30T21:43:44.219 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.215+0000 7f42fb7fe700 1 -- 172.21.15.27:0/353172590 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f430802a070 con 0x7f43140fab90 2023-11-30T21:43:44.219 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.216+0000 7f42fb7fe700 1 --2- 172.21.15.27:0/353172590 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f430002fd50 0x7f4300032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:44.220 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.216+0000 7f42fb7fe700 1 -- 172.21.15.27:0/353172590 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4308042930 con 0x7f43140fab90 2023-11-30T21:43:44.220 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.216+0000 7f43127fc700 1 --2- 172.21.15.27:0/353172590 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f430002fd50 0x7f4300032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:44.220 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.216+0000 7f431948a700 1 -- 172.21.15.27:0/353172590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4314059100 con 0x7f43140fab90 2023-11-30T21:43:44.220 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.216+0000 7f43127fc700 1 --2- 172.21.15.27:0/353172590 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f430002fd50 0x7f4300032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f43040096a0 tx=0x7f4304006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:44.221 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.220+0000 7f42fb7fe700 1 -- 172.21.15.27:0/353172590 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4308014070 con 0x7f43140fab90 2023-11-30T21:43:44.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:44.417+0000 7f431948a700 1 -- 172.21.15.27:0/353172590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f4314045ad0 con 0x7f43140fab90 2023-11-30T21:43:44.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:44 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:43.356562+0000 mgr.smithi027.brrjcl (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-11-30T21:43:44.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:44 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:43:43.357165+0000 mgr.smithi027.brrjcl (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2023-11-30T21:43:44.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:44 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:43.359639+0000 mon.smithi027 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.27:0/2955012891' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:45.364 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:45.362+0000 7f42fb7fe700 1 -- 172.21.15.27:0/353172590 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f4308016030 con 0x7f43140fab90 2023-11-30T21:43:45.364 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:45.363+0000 7f42fb7fe700 1 -- 172.21.15.27:0/353172590 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v8) v1 ==== 89+0+0 (secure 0 0 0) 0x7f4308017880 con 0x7f43140fab90 2023-11-30T21:43:45.368 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:45.365+0000 7f431948a700 1 -- 172.21.15.27:0/353172590 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f430002fd50 msgr2=0x7f4300032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:45.369 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:45.365+0000 7f431948a700 1 --2- 172.21.15.27:0/353172590 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f430002fd50 0x7f4300032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f43040096a0 tx=0x7f4304006b40).stop 2023-11-30T21:43:45.369 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:45.365+0000 7f431948a700 1 -- 172.21.15.27:0/353172590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140fab90 msgr2=0x7f431410cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:45.369 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:45.365+0000 7f431948a700 1 --2- 172.21.15.27:0/353172590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140fab90 0x7f431410cb40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f4308006a40 tx=0x7f4308006f70).stop 2023-11-30T21:43:45.369 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:45.365+0000 7f431948a700 1 -- 172.21.15.27:0/353172590 shutdown_connections 2023-11-30T21:43:45.369 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:45.365+0000 7f431948a700 1 --2- 172.21.15.27:0/353172590 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7f430002fd50 0x7f4300032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:45.369 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:45.365+0000 7f431948a700 1 --2- 172.21.15.27:0/353172590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43140fab90 0x7f431410cb40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:45.369 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:45.365+0000 7f431948a700 1 -- 172.21.15.27:0/353172590 >> 172.21.15.27:0/353172590 conn(0x7f43140f63b0 msgr2=0x7f43140ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:45.369 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:45.366+0000 7f431948a700 1 -- 172.21.15.27:0/353172590 shutdown_connections 2023-11-30T21:43:45.369 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:45.366+0000 7f431948a700 1 -- 172.21.15.27:0/353172590 wait complete. 2023-11-30T21:43:45.705 INFO:teuthology.orchestra.run.smithi027.stderr:Deploying prometheus service with default placement... 2023-11-30T21:43:45.705 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-11-30T21:43:45.706 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:45 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:44.418431+0000 mon.smithi027 (mon.0) 58 : audit [INF] from='client.? 172.21.15.27:0/353172590' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-11-30T21:43:46.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:46 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:45.364730+0000 mon.smithi027 (mon.0) 59 : audit [INF] from='client.? 172.21.15.27:0/353172590' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-11-30T21:43:46.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:46 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:45.364801+0000 mon.smithi027 (mon.0) 60 : cluster [DBG] mgrmap e8: smithi027.brrjcl(active, since 16s) 2023-11-30T21:43:46.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.521+0000 7facc285b700 1 Processor -- start 2023-11-30T21:43:46.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.521+0000 7facc285b700 1 -- start start 2023-11-30T21:43:46.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.521+0000 7facc285b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc06fb70 0x7facbc100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:46.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.521+0000 7facc285b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc0fc770 con 0x7facbc06fb70 2023-11-30T21:43:46.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.521+0000 7facbbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc06fb70 0x7facbc100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:46.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.521+0000 7facbbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc06fb70 0x7facbc100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48478/0 (socket says 172.21.15.27:48478) 2023-11-30T21:43:46.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.521+0000 7facbbfff700 1 -- 172.21.15.27:0/2236392222 learned_addr learned my addr 172.21.15.27:0/2236392222 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:46.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.522+0000 7facbbfff700 1 -- 172.21.15.27:0/2236392222 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc101270 con 0x7facbc06fb70 2023-11-30T21:43:46.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.522+0000 7facbbfff700 1 --2- 172.21.15.27:0/2236392222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc06fb70 0x7facbc100d80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7facac00b5f0 tx=0x7facac006ee0).ready entity=mon.0 client_cookie=cab78770035ed6cd server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:46.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.522+0000 7facbaffd700 1 -- 172.21.15.27:0/2236392222 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facac00d600 con 0x7facbc06fb70 2023-11-30T21:43:46.524 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.522+0000 7facbaffd700 1 -- 172.21.15.27:0/2236392222 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7facac00dbd0 con 0x7facbc06fb70 2023-11-30T21:43:46.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.522+0000 7facbaffd700 1 -- 172.21.15.27:0/2236392222 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facac011c00 con 0x7facbc06fb70 2023-11-30T21:43:46.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.523+0000 7facc285b700 1 -- 172.21.15.27:0/2236392222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc06fb70 msgr2=0x7facbc100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:46.525 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.523+0000 7facc285b700 1 --2- 172.21.15.27:0/2236392222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc06fb70 0x7facbc100d80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7facac00b5f0 tx=0x7facac006ee0).stop 2023-11-30T21:43:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.523+0000 7facc285b700 1 -- 172.21.15.27:0/2236392222 shutdown_connections 2023-11-30T21:43:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.523+0000 7facc285b700 1 --2- 172.21.15.27:0/2236392222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc06fb70 0x7facbc100d80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.523+0000 7facc285b700 1 -- 172.21.15.27:0/2236392222 >> 172.21.15.27:0/2236392222 conn(0x7facbc0f63a0 msgr2=0x7facbc0f87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.523+0000 7facc285b700 1 -- 172.21.15.27:0/2236392222 shutdown_connections 2023-11-30T21:43:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.524+0000 7facc285b700 1 -- 172.21.15.27:0/2236392222 wait complete. 2023-11-30T21:43:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.524+0000 7facc285b700 1 Processor -- start 2023-11-30T21:43:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.524+0000 7facc285b700 1 -- start start 2023-11-30T21:43:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.524+0000 7facc285b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc06fb70 0x7facbc10cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.524+0000 7facc285b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc10d0c0 con 0x7facbc06fb70 2023-11-30T21:43:46.527 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.525+0000 7facbbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc06fb70 0x7facbc10cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.525+0000 7facbbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc06fb70 0x7facbc10cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48492/0 (socket says 172.21.15.27:48492) 2023-11-30T21:43:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.525+0000 7facbbfff700 1 -- 172.21.15.27:0/4086066654 learned_addr learned my addr 172.21.15.27:0/4086066654 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.525+0000 7facbbfff700 1 -- 172.21.15.27:0/4086066654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac00b200 con 0x7facbc06fb70 2023-11-30T21:43:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.525+0000 7facbbfff700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc06fb70 0x7facbc10cbd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7facac00b190 tx=0x7facac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.525+0000 7facb8ff9700 1 -- 172.21.15.27:0/4086066654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facac01b410 con 0x7facbc06fb70 2023-11-30T21:43:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.525+0000 7facb8ff9700 1 -- 172.21.15.27:0/4086066654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7facac01b9e0 con 0x7facbc06fb70 2023-11-30T21:43:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.525+0000 7facc285b700 1 -- 172.21.15.27:0/4086066654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facbc10d2c0 con 0x7facbc06fb70 2023-11-30T21:43:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.525+0000 7facb8ff9700 1 -- 172.21.15.27:0/4086066654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facac01a880 con 0x7facbc06fb70 2023-11-30T21:43:46.528 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.525+0000 7facc285b700 1 -- 172.21.15.27:0/4086066654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facbc10d6d0 con 0x7facbc06fb70 2023-11-30T21:43:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.526+0000 7facb8ff9700 1 -- 172.21.15.27:0/4086066654 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7facac00f070 con 0x7facbc06fb70 2023-11-30T21:43:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.526+0000 7facc285b700 1 -- 172.21.15.27:0/4086066654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facbc045ad0 con 0x7facbc06fb70 2023-11-30T21:43:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.527+0000 7facb8ff9700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7faca802fe00 0x7faca80322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:46.529 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.527+0000 7facb8ff9700 1 -- 172.21.15.27:0/4086066654 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7facac041260 con 0x7facbc06fb70 2023-11-30T21:43:46.530 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.527+0000 7facbb7fe700 1 -- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7faca802fe00 msgr2=0x7faca80322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2518541463 2023-11-30T21:43:46.530 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.527+0000 7facbb7fe700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7faca802fe00 0x7faca80322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-30T21:43:46.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.531+0000 7facb8ff9700 1 -- 172.21.15.27:0/4086066654 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7facac022d20 con 0x7facbc06fb70 2023-11-30T21:43:46.729 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.728+0000 7facbb7fe700 1 -- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7faca802fe00 msgr2=0x7faca80322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2518541463 2023-11-30T21:43:46.729 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.728+0000 7facbb7fe700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7faca802fe00 0x7faca80322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-30T21:43:46.764 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:46.763+0000 7facc285b700 1 -- 172.21.15.27:0/4086066654 --> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7facbc10dfb0 con 0x7faca802fe00 2023-11-30T21:43:47.130 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:47.129+0000 7facbb7fe700 1 -- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7faca802fe00 msgr2=0x7faca80322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2518541463 2023-11-30T21:43:47.130 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:47.129+0000 7facbb7fe700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7faca802fe00 0x7faca80322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-30T21:43:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:47.930+0000 7facbb7fe700 1 -- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7faca802fe00 msgr2=0x7faca80322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2518541463 2023-11-30T21:43:47.931 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:47.930+0000 7facbb7fe700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7faca802fe00 0x7faca80322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-30T21:43:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:49.531+0000 7facbb7fe700 1 -- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7faca802fe00 msgr2=0x7faca80322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2518541463 2023-11-30T21:43:49.532 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:49.531+0000 7facbb7fe700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7faca802fe00 0x7faca80322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-30T21:43:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:50.607+0000 7facb8ff9700 1 -- 172.21.15.27:0/4086066654 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 9) v1 ==== 35984+0+0 (secure 0 0 0) 0x7facac0257e0 con 0x7facbc06fb70 2023-11-30T21:43:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:50.607+0000 7facb8ff9700 1 -- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7faca802fe00 msgr2=0x7faca80322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-30T21:43:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:50.607+0000 7facb8ff9700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7faca802fe00 0x7faca80322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-30T21:43:51.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:50 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:50.557054+0000 mon.smithi027 (mon.0) 61 : cluster [INF] Active manager daemon smithi027.brrjcl restarted 2023-11-30T21:43:51.012 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:50 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:50.557433+0000 mon.smithi027 (mon.0) 62 : cluster [INF] Activating manager daemon smithi027.brrjcl 2023-11-30T21:43:51.012 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:50 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:50.559148+0000 mon.smithi027 (mon.0) 63 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-11-30T21:43:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.613+0000 7facb8ff9700 1 -- 172.21.15.27:0/4086066654 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7facac03da30 con 0x7facbc06fb70 2023-11-30T21:43:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.613+0000 7facb8ff9700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7faca8033c20 0x7faca8036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.613+0000 7facb8ff9700 1 -- 172.21.15.27:0/4086066654 --> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7facbc10dfb0 con 0x7faca8033c20 2023-11-30T21:43:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.614+0000 7facbb7fe700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7faca8033c20 0x7faca8036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.614+0000 7facbb7fe700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7faca8033c20 0x7faca8036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7facb0006fd0 tx=0x7facb0006d30).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.619+0000 7facb8ff9700 1 -- 172.21.15.27:0/4086066654 <== mgr.14142 v2:172.21.15.27:6800/2420177321 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7facbc10dfb0 con 0x7faca8033c20 2023-11-30T21:43:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-11-30T21:43:51.623 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.621+0000 7facc285b700 1 -- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7faca8033c20 msgr2=0x7faca8036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:51.623 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.621+0000 7facc285b700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7faca8033c20 0x7faca8036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7facb0006fd0 tx=0x7facb0006d30).stop 2023-11-30T21:43:51.623 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.621+0000 7facc285b700 1 -- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc06fb70 msgr2=0x7facbc10cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:51.623 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.621+0000 7facc285b700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc06fb70 0x7facbc10cbd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7facac00b190 tx=0x7facac006ee0).stop 2023-11-30T21:43:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.621+0000 7facc285b700 1 -- 172.21.15.27:0/4086066654 shutdown_connections 2023-11-30T21:43:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.621+0000 7facc285b700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc06fb70 0x7facbc10cbd0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.621+0000 7facc285b700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7faca8033c20 0x7faca8036010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.621+0000 7facc285b700 1 --2- 172.21.15.27:0/4086066654 >> [v2:172.21.15.27:6800/2518541463,v1:172.21.15.27:6801/2518541463] conn(0x7faca802fe00 0x7faca80322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-30T21:43:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.621+0000 7facc285b700 1 -- 172.21.15.27:0/4086066654 >> 172.21.15.27:0/4086066654 conn(0x7facbc0f63a0 msgr2=0x7facbc0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.621+0000 7facc285b700 1 -- 172.21.15.27:0/4086066654 shutdown_connections 2023-11-30T21:43:51.625 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:51.621+0000 7facc285b700 1 -- 172.21.15.27:0/4086066654 wait complete. 2023-11-30T21:43:52.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:50.609873+0000 mon.smithi027 (mon.0) 64 : cluster [DBG] mgrmap e9: smithi027.brrjcl(active, starting, since 0.0525013s) 2023-11-30T21:43:52.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:50.612671+0000 mon.smithi027 (mon.0) 65 : audit [DBG] from='mgr.14142 172.21.15.27:0/1800026764' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2023-11-30T21:43:52.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:50.612868+0000 mon.smithi027 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.27:0/1800026764' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mgr metadata", "who": "smithi027.brrjcl", "id": "smithi027.brrjcl"}]: dispatch 2023-11-30T21:43:52.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:50.613048+0000 mon.smithi027 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.27:0/1800026764' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-30T21:43:52.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:50.613193+0000 mon.smithi027 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.27:0/1800026764' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-30T21:43:52.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:50.613318+0000 mon.smithi027 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.27:0/1800026764' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-30T21:43:52.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:50.627289+0000 mon.smithi027 (mon.0) 70 : cluster [INF] Manager daemon smithi027.brrjcl is now available 2023-11-30T21:43:52.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:50.730193+0000 mon.smithi027 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.27:0/1800026764' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-30T21:43:52.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:50.731788+0000 mon.smithi027 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.27:0/1800026764' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-30T21:43:52.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:50.748599+0000 mon.smithi027 (mon.0) 73 : audit [INF] from='mgr.14142 172.21.15.27:0/1800026764' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/mirror_snapshot_schedule"}]: dispatch 2023-11-30T21:43:52.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:50.757094+0000 mon.smithi027 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.27:0/1800026764' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/trash_purge_schedule"}]: dispatch 2023-11-30T21:43:52.011 INFO:teuthology.orchestra.run.smithi027.stderr:Deploying grafana service with default placement... 2023-11-30T21:43:52.012 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-11-30T21:43:52.381 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.378+0000 7f1a9c728700 1 Processor -- start 2023-11-30T21:43:52.381 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.378+0000 7f1a9c728700 1 -- start start 2023-11-30T21:43:52.382 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.378+0000 7f1a9c728700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940703d0 0x7f1a94100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:52.382 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.378+0000 7f1a9c728700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a940fc730 con 0x7f1a940703d0 2023-11-30T21:43:52.382 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.378+0000 7f1a9a4c4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940703d0 0x7f1a94100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:52.382 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.378+0000 7f1a9a4c4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940703d0 0x7f1a94100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48546/0 (socket says 172.21.15.27:48546) 2023-11-30T21:43:52.382 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.378+0000 7f1a9a4c4700 1 -- 172.21.15.27:0/1731730335 learned_addr learned my addr 172.21.15.27:0/1731730335 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:52.382 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.378+0000 7f1a9a4c4700 1 -- 172.21.15.27:0/1731730335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a94101230 con 0x7f1a940703d0 2023-11-30T21:43:52.382 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.378+0000 7f1a9a4c4700 1 --2- 172.21.15.27:0/1731730335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940703d0 0x7f1a94100d40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f1a9000b5f0 tx=0x7f1a90006ee0).ready entity=mon.0 client_cookie=44147a4f28831444 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:52.382 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.379+0000 7f1a994c2700 1 -- 172.21.15.27:0/1731730335 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9000d600 con 0x7f1a940703d0 2023-11-30T21:43:52.382 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.379+0000 7f1a994c2700 1 -- 172.21.15.27:0/1731730335 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1a9000dbd0 con 0x7f1a940703d0 2023-11-30T21:43:52.382 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.379+0000 7f1a994c2700 1 -- 172.21.15.27:0/1731730335 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a90011c00 con 0x7f1a940703d0 2023-11-30T21:43:52.384 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.379+0000 7f1a9c728700 1 -- 172.21.15.27:0/1731730335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940703d0 msgr2=0x7f1a94100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:52.385 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.379+0000 7f1a9c728700 1 --2- 172.21.15.27:0/1731730335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940703d0 0x7f1a94100d40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f1a9000b5f0 tx=0x7f1a90006ee0).stop 2023-11-30T21:43:52.385 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.379+0000 7f1a9c728700 1 -- 172.21.15.27:0/1731730335 shutdown_connections 2023-11-30T21:43:52.385 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.379+0000 7f1a9c728700 1 --2- 172.21.15.27:0/1731730335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940703d0 0x7f1a94100d40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:52.385 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.379+0000 7f1a9c728700 1 -- 172.21.15.27:0/1731730335 >> 172.21.15.27:0/1731730335 conn(0x7f1a940f63c0 msgr2=0x7f1a940f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:52.385 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.379+0000 7f1a9c728700 1 -- 172.21.15.27:0/1731730335 shutdown_connections 2023-11-30T21:43:52.385 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.379+0000 7f1a9c728700 1 -- 172.21.15.27:0/1731730335 wait complete. 2023-11-30T21:43:52.385 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.380+0000 7f1a9c728700 1 Processor -- start 2023-11-30T21:43:52.386 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.380+0000 7f1a9c728700 1 -- start start 2023-11-30T21:43:52.386 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.380+0000 7f1a9c728700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940703d0 0x7f1a9410cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:52.386 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.380+0000 7f1a9c728700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9410d0a0 con 0x7f1a940703d0 2023-11-30T21:43:52.386 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.380+0000 7f1a9a4c4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940703d0 0x7f1a9410cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:52.386 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.380+0000 7f1a9a4c4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940703d0 0x7f1a9410cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48552/0 (socket says 172.21.15.27:48552) 2023-11-30T21:43:52.386 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.380+0000 7f1a9a4c4700 1 -- 172.21.15.27:0/952088107 learned_addr learned my addr 172.21.15.27:0/952088107 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:52.386 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.380+0000 7f1a9a4c4700 1 -- 172.21.15.27:0/952088107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9000b200 con 0x7f1a940703d0 2023-11-30T21:43:52.387 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.381+0000 7f1a9a4c4700 1 --2- 172.21.15.27:0/952088107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940703d0 0x7f1a9410cbb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f1a9000b190 tx=0x7f1a90006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:52.387 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.381+0000 7f1a86ffd700 1 -- 172.21.15.27:0/952088107 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9001b410 con 0x7f1a940703d0 2023-11-30T21:43:52.387 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.381+0000 7f1a86ffd700 1 -- 172.21.15.27:0/952088107 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1a9001b9e0 con 0x7f1a940703d0 2023-11-30T21:43:52.387 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.381+0000 7f1a9c728700 1 -- 172.21.15.27:0/952088107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a9410d2a0 con 0x7f1a940703d0 2023-11-30T21:43:52.387 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.381+0000 7f1a86ffd700 1 -- 172.21.15.27:0/952088107 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9001a880 con 0x7f1a940703d0 2023-11-30T21:43:52.387 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.381+0000 7f1a9c728700 1 -- 172.21.15.27:0/952088107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a9410d6b0 con 0x7f1a940703d0 2023-11-30T21:43:52.387 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.382+0000 7f1a86ffd700 1 -- 172.21.15.27:0/952088107 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f1a9000f070 con 0x7f1a940703d0 2023-11-30T21:43:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.382+0000 7f1a86ffd700 1 --2- 172.21.15.27:0/952088107 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f1a7c02fcf0 0x7f1a7c0321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.382+0000 7f1a86ffd700 1 -- 172.21.15.27:0/952088107 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f1a900412d0 con 0x7f1a940703d0 2023-11-30T21:43:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.382+0000 7f1a99cc3700 1 --2- 172.21.15.27:0/952088107 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f1a7c02fcf0 0x7f1a7c0321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.382+0000 7f1a9c728700 1 -- 172.21.15.27:0/952088107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a94059100 con 0x7f1a940703d0 2023-11-30T21:43:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.382+0000 7f1a99cc3700 1 --2- 172.21.15.27:0/952088107 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f1a7c02fcf0 0x7f1a7c0321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1a880096a0 tx=0x7f1a88006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:52.388 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.387+0000 7f1a86ffd700 1 -- 172.21.15.27:0/952088107 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a90016070 con 0x7f1a940703d0 2023-11-30T21:43:52.578 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.575+0000 7f1a9c728700 1 -- 172.21.15.27:0/952088107 --> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a94106a60 con 0x7f1a7c02fcf0 2023-11-30T21:43:52.581 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.579+0000 7f1a86ffd700 1 -- 172.21.15.27:0/952088107 <== mgr.14142 v2:172.21.15.27:6800/2420177321 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f1a94106a60 con 0x7f1a7c02fcf0 2023-11-30T21:43:52.581 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-11-30T21:43:52.583 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.580+0000 7f1a9c728700 1 -- 172.21.15.27:0/952088107 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f1a7c02fcf0 msgr2=0x7f1a7c0321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:52.583 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.580+0000 7f1a9c728700 1 --2- 172.21.15.27:0/952088107 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f1a7c02fcf0 0x7f1a7c0321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f1a880096a0 tx=0x7f1a88006b40).stop 2023-11-30T21:43:52.583 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.580+0000 7f1a9c728700 1 -- 172.21.15.27:0/952088107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940703d0 msgr2=0x7f1a9410cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:52.583 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.580+0000 7f1a9c728700 1 --2- 172.21.15.27:0/952088107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940703d0 0x7f1a9410cbb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f1a9000b190 tx=0x7f1a90006ee0).stop 2023-11-30T21:43:52.583 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.580+0000 7f1a9c728700 1 -- 172.21.15.27:0/952088107 shutdown_connections 2023-11-30T21:43:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.580+0000 7f1a9c728700 1 --2- 172.21.15.27:0/952088107 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f1a7c02fcf0 0x7f1a7c0321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.580+0000 7f1a9c728700 1 --2- 172.21.15.27:0/952088107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a940703d0 0x7f1a9410cbb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.580+0000 7f1a9c728700 1 -- 172.21.15.27:0/952088107 >> 172.21.15.27:0/952088107 conn(0x7f1a940f63c0 msgr2=0x7f1a940f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.581+0000 7f1a9c728700 1 -- 172.21.15.27:0/952088107 shutdown_connections 2023-11-30T21:43:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:52.581+0000 7f1a9c728700 1 -- 172.21.15.27:0/952088107 wait complete. 2023-11-30T21:43:52.894 INFO:teuthology.orchestra.run.smithi027.stderr:Deploying node-exporter service with default placement... 2023-11-30T21:43:52.895 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-11-30T21:43:52.897 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:52 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:51.617361+0000 mon.smithi027 (mon.0) 75 : cluster [DBG] mgrmap e10: smithi027.brrjcl(active, since 1.06s) 2023-11-30T21:43:52.897 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:52 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:43:51.618600+0000 mgr.smithi027.brrjcl (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-11-30T21:43:52.897 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:52 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:51.620660+0000 mon.smithi027 (mon.0) 76 : audit [INF] from='mgr.14142 172.21.15.27:0/1800026764' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:52.897 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:52 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:52.580731+0000 mon.smithi027 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.27:0/1800026764' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:53.229 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.226+0000 7f28b7786700 1 Processor -- start 2023-11-30T21:43:53.229 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.226+0000 7f28b7786700 1 -- start start 2023-11-30T21:43:53.229 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.226+0000 7f28b7786700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b00703f0 0x7f28b00fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:53.229 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.226+0000 7f28b7786700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b00fee10 con 0x7f28b00703f0 2023-11-30T21:43:53.229 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.227+0000 7f28b5522700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b00703f0 0x7f28b00fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:53.229 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.227+0000 7f28b5522700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b00703f0 0x7f28b00fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36666/0 (socket says 172.21.15.27:36666) 2023-11-30T21:43:53.229 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.227+0000 7f28b5522700 1 -- 172.21.15.27:0/961778591 learned_addr learned my addr 172.21.15.27:0/961778591 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:53.230 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.227+0000 7f28b5522700 1 -- 172.21.15.27:0/961778591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b00fef50 con 0x7f28b00703f0 2023-11-30T21:43:53.230 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.227+0000 7f28b5522700 1 --2- 172.21.15.27:0/961778591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b00703f0 0x7f28b00fe920 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f28a400b5f0 tx=0x7f28a4006ee0).ready entity=mon.0 client_cookie=225bfa24ff97e43d server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:53.230 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.227+0000 7f28a3fff700 1 -- 172.21.15.27:0/961778591 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28a400d600 con 0x7f28b00703f0 2023-11-30T21:43:53.230 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.227+0000 7f28a3fff700 1 -- 172.21.15.27:0/961778591 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f28a400dbd0 con 0x7f28b00703f0 2023-11-30T21:43:53.230 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.227+0000 7f28a3fff700 1 -- 172.21.15.27:0/961778591 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28a4011c00 con 0x7f28b00703f0 2023-11-30T21:43:53.230 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.228+0000 7f28b7786700 1 -- 172.21.15.27:0/961778591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b00703f0 msgr2=0x7f28b00fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:53.230 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.228+0000 7f28b7786700 1 --2- 172.21.15.27:0/961778591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b00703f0 0x7f28b00fe920 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f28a400b5f0 tx=0x7f28a4006ee0).stop 2023-11-30T21:43:53.232 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.228+0000 7f28b7786700 1 -- 172.21.15.27:0/961778591 shutdown_connections 2023-11-30T21:43:53.232 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.228+0000 7f28b7786700 1 --2- 172.21.15.27:0/961778591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b00703f0 0x7f28b00fe920 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:53.232 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.228+0000 7f28b7786700 1 -- 172.21.15.27:0/961778591 >> 172.21.15.27:0/961778591 conn(0x7f28b00f63c0 msgr2=0x7f28b00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:53.232 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.228+0000 7f28b7786700 1 -- 172.21.15.27:0/961778591 shutdown_connections 2023-11-30T21:43:53.232 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.228+0000 7f28b7786700 1 -- 172.21.15.27:0/961778591 wait complete. 2023-11-30T21:43:53.232 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.229+0000 7f28b7786700 1 Processor -- start 2023-11-30T21:43:53.232 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.229+0000 7f28b7786700 1 -- start start 2023-11-30T21:43:53.233 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.229+0000 7f28b7786700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b00703f0 0x7f28b010aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:53.233 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.229+0000 7f28b7786700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b010af40 con 0x7f28b00703f0 2023-11-30T21:43:53.233 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.229+0000 7f28b5522700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b00703f0 0x7f28b010aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:53.233 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.229+0000 7f28b5522700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b00703f0 0x7f28b010aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36680/0 (socket says 172.21.15.27:36680) 2023-11-30T21:43:53.233 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.229+0000 7f28b5522700 1 -- 172.21.15.27:0/3337974657 learned_addr learned my addr 172.21.15.27:0/3337974657 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:53.233 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.229+0000 7f28b5522700 1 -- 172.21.15.27:0/3337974657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a400b200 con 0x7f28b00703f0 2023-11-30T21:43:53.233 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.229+0000 7f28b5522700 1 --2- 172.21.15.27:0/3337974657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b00703f0 0x7f28b010aa50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f28a400e040 tx=0x7f28a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:53.233 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.230+0000 7f28a1ffb700 1 -- 172.21.15.27:0/3337974657 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28a400d9f0 con 0x7f28b00703f0 2023-11-30T21:43:53.233 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.230+0000 7f28b7786700 1 -- 172.21.15.27:0/3337974657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b010b140 con 0x7f28b00703f0 2023-11-30T21:43:53.233 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.230+0000 7f28a1ffb700 1 -- 172.21.15.27:0/3337974657 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f28a401a9f0 con 0x7f28b00703f0 2023-11-30T21:43:53.233 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.230+0000 7f28a1ffb700 1 -- 172.21.15.27:0/3337974657 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28a401b8c0 con 0x7f28b00703f0 2023-11-30T21:43:53.233 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.230+0000 7f28b7786700 1 -- 172.21.15.27:0/3337974657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b010b550 con 0x7f28b00703f0 2023-11-30T21:43:53.233 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.231+0000 7f28a1ffb700 1 -- 172.21.15.27:0/3337974657 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f28a400f070 con 0x7f28b00703f0 2023-11-30T21:43:53.234 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.231+0000 7f28a1ffb700 1 --2- 172.21.15.27:0/3337974657 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f289802fe60 0x7f2898032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:53.234 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.231+0000 7f28b7786700 1 -- 172.21.15.27:0/3337974657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b0045ad0 con 0x7f28b00703f0 2023-11-30T21:43:53.234 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.231+0000 7f28a1ffb700 1 -- 172.21.15.27:0/3337974657 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f28a4041570 con 0x7f28b00703f0 2023-11-30T21:43:53.234 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.231+0000 7f28b4d21700 1 --2- 172.21.15.27:0/3337974657 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f289802fe60 0x7f2898032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:53.234 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.231+0000 7f28b4d21700 1 --2- 172.21.15.27:0/3337974657 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f289802fe60 0x7f2898032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f28ac0096a0 tx=0x7f28ac006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:53.236 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.235+0000 7f28a1ffb700 1 -- 172.21.15.27:0/3337974657 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f28a4016020 con 0x7f28b00703f0 2023-11-30T21:43:53.423 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.422+0000 7f28b7786700 1 -- 172.21.15.27:0/3337974657 --> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f28b010be20 con 0x7f289802fe60 2023-11-30T21:43:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.426+0000 7f28a1ffb700 1 -- 172.21.15.27:0/3337974657 <== mgr.14142 v2:172.21.15.27:6800/2420177321 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f28b010be20 con 0x7f289802fe60 2023-11-30T21:43:53.427 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-11-30T21:43:53.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.428+0000 7f28b7786700 1 -- 172.21.15.27:0/3337974657 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f289802fe60 msgr2=0x7f2898032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:53.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.428+0000 7f28b7786700 1 --2- 172.21.15.27:0/3337974657 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f289802fe60 0x7f2898032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f28ac0096a0 tx=0x7f28ac006b40).stop 2023-11-30T21:43:53.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.428+0000 7f28b7786700 1 -- 172.21.15.27:0/3337974657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b00703f0 msgr2=0x7f28b010aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:53.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.428+0000 7f28b7786700 1 --2- 172.21.15.27:0/3337974657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b00703f0 0x7f28b010aa50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f28a400e040 tx=0x7f28a4006ee0).stop 2023-11-30T21:43:53.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.428+0000 7f28b7786700 1 -- 172.21.15.27:0/3337974657 shutdown_connections 2023-11-30T21:43:53.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.428+0000 7f28b7786700 1 --2- 172.21.15.27:0/3337974657 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f289802fe60 0x7f2898032320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:53.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.428+0000 7f28b7786700 1 --2- 172.21.15.27:0/3337974657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28b00703f0 0x7f28b010aa50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:53.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.428+0000 7f28b7786700 1 -- 172.21.15.27:0/3337974657 >> 172.21.15.27:0/3337974657 conn(0x7f28b00f63c0 msgr2=0x7f28b00f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:53.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.428+0000 7f28b7786700 1 -- 172.21.15.27:0/3337974657 shutdown_connections 2023-11-30T21:43:53.430 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:53.428+0000 7f28b7786700 1 -- 172.21.15.27:0/3337974657 wait complete. 2023-11-30T21:43:53.760 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:53 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:52.577993+0000 mgr.smithi027.brrjcl (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-11-30T21:43:53.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:53 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:43:52.578512+0000 mgr.smithi027.brrjcl (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-11-30T21:43:53.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:53 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:52.621681+0000 mon.smithi027 (mon.0) 78 : cluster [DBG] mgrmap e11: smithi027.brrjcl(active, since 2s) 2023-11-30T21:43:53.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:53 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:53.427011+0000 mon.smithi027 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.27:0/1800026764' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:53.811 INFO:teuthology.orchestra.run.smithi027.stderr:Deploying alertmanager service with default placement... 2023-11-30T21:43:53.811 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-11-30T21:43:54.170 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.168+0000 7f4e90aea700 1 Processor -- start 2023-11-30T21:43:54.170 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.168+0000 7f4e90aea700 1 -- start start 2023-11-30T21:43:54.170 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.169+0000 7f4e90aea700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e8c0fab80 0x7f4e8c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:54.170 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.169+0000 7f4e90aea700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c0fb490 con 0x7f4e8c0fab80 2023-11-30T21:43:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.169+0000 7f4e8a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e8c0fab80 0x7f4e8c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.169+0000 7f4e8a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e8c0fab80 0x7f4e8c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36686/0 (socket says 172.21.15.27:36686) 2023-11-30T21:43:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.169+0000 7f4e8a59c700 1 -- 172.21.15.27:0/2690674683 learned_addr learned my addr 172.21.15.27:0/2690674683 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.169+0000 7f4e8a59c700 1 -- 172.21.15.27:0/2690674683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c0fb5d0 con 0x7f4e8c0fab80 2023-11-30T21:43:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.169+0000 7f4e8a59c700 1 --2- 172.21.15.27:0/2690674683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e8c0fab80 0x7f4e8c0fafa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f4e7400b5f0 tx=0x7f4e74006ee0).ready entity=mon.0 client_cookie=1b4e51b77874e334 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.169+0000 7f4e8959a700 1 -- 172.21.15.27:0/2690674683 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e7400d600 con 0x7f4e8c0fab80 2023-11-30T21:43:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.169+0000 7f4e8959a700 1 -- 172.21.15.27:0/2690674683 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4e7400dbd0 con 0x7f4e8c0fab80 2023-11-30T21:43:54.173 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.169+0000 7f4e8959a700 1 -- 172.21.15.27:0/2690674683 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e74011c00 con 0x7f4e8c0fab80 2023-11-30T21:43:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.170+0000 7f4e90aea700 1 -- 172.21.15.27:0/2690674683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e8c0fab80 msgr2=0x7f4e8c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.170+0000 7f4e90aea700 1 --2- 172.21.15.27:0/2690674683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e8c0fab80 0x7f4e8c0fafa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f4e7400b5f0 tx=0x7f4e74006ee0).stop 2023-11-30T21:43:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.170+0000 7f4e90aea700 1 -- 172.21.15.27:0/2690674683 shutdown_connections 2023-11-30T21:43:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.170+0000 7f4e90aea700 1 --2- 172.21.15.27:0/2690674683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e8c0fab80 0x7f4e8c0fafa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.170+0000 7f4e90aea700 1 -- 172.21.15.27:0/2690674683 >> 172.21.15.27:0/2690674683 conn(0x7f4e8c0f63c0 msgr2=0x7f4e8c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.170+0000 7f4e90aea700 1 -- 172.21.15.27:0/2690674683 shutdown_connections 2023-11-30T21:43:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.170+0000 7f4e90aea700 1 -- 172.21.15.27:0/2690674683 wait complete. 2023-11-30T21:43:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.171+0000 7f4e90aea700 1 Processor -- start 2023-11-30T21:43:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.171+0000 7f4e90aea700 1 -- start start 2023-11-30T21:43:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.171+0000 7f4e90aea700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e8c0fab80 0x7f4e8c10aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.171+0000 7f4e90aea700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c10af00 con 0x7f4e8c0fab80 2023-11-30T21:43:54.174 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.171+0000 7f4e8a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e8c0fab80 0x7f4e8c10aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:54.175 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.171+0000 7f4e8a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e8c0fab80 0x7f4e8c10aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36696/0 (socket says 172.21.15.27:36696) 2023-11-30T21:43:54.175 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.171+0000 7f4e8a59c700 1 -- 172.21.15.27:0/984783076 learned_addr learned my addr 172.21.15.27:0/984783076 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:54.175 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.171+0000 7f4e8a59c700 1 -- 172.21.15.27:0/984783076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7400b200 con 0x7f4e8c0fab80 2023-11-30T21:43:54.175 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.171+0000 7f4e8a59c700 1 --2- 172.21.15.27:0/984783076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e8c0fab80 0x7f4e8c10aa10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f4e7400e040 tx=0x7f4e74006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:54.175 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.172+0000 7f4e82ffd700 1 -- 172.21.15.27:0/984783076 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e7401b410 con 0x7f4e8c0fab80 2023-11-30T21:43:54.175 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.172+0000 7f4e90aea700 1 -- 172.21.15.27:0/984783076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8c10b100 con 0x7f4e8c0fab80 2023-11-30T21:43:54.175 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.172+0000 7f4e82ffd700 1 -- 172.21.15.27:0/984783076 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4e7401b9e0 con 0x7f4e8c0fab80 2023-11-30T21:43:54.175 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.172+0000 7f4e82ffd700 1 -- 172.21.15.27:0/984783076 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e7401a820 con 0x7f4e8c0fab80 2023-11-30T21:43:54.175 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.172+0000 7f4e90aea700 1 -- 172.21.15.27:0/984783076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8c10b510 con 0x7f4e8c0fab80 2023-11-30T21:43:54.176 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.172+0000 7f4e82ffd700 1 -- 172.21.15.27:0/984783076 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f4e7400f070 con 0x7f4e8c0fab80 2023-11-30T21:43:54.176 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.173+0000 7f4e90aea700 1 -- 172.21.15.27:0/984783076 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8c045ad0 con 0x7f4e8c0fab80 2023-11-30T21:43:54.176 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.173+0000 7f4e82ffd700 1 --2- 172.21.15.27:0/984783076 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f4e6c02fe60 0x7f4e6c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:54.176 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.173+0000 7f4e82ffd700 1 -- 172.21.15.27:0/984783076 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4e740415a0 con 0x7f4e8c0fab80 2023-11-30T21:43:54.176 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.173+0000 7f4e89d9b700 1 --2- 172.21.15.27:0/984783076 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f4e6c02fe60 0x7f4e6c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:54.176 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.173+0000 7f4e89d9b700 1 --2- 172.21.15.27:0/984783076 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f4e6c02fe60 0x7f4e6c032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f4e7c0096a0 tx=0x7f4e7c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:54.178 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.177+0000 7f4e82ffd700 1 -- 172.21.15.27:0/984783076 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e74022310 con 0x7f4e8c0fab80 2023-11-30T21:43:54.367 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.365+0000 7f4e90aea700 1 -- 172.21.15.27:0/984783076 --> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e8c10bdf0 con 0x7f4e6c02fe60 2023-11-30T21:43:54.370 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.369+0000 7f4e82ffd700 1 -- 172.21.15.27:0/984783076 <== mgr.14142 v2:172.21.15.27:6800/2420177321 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f4e8c10bdf0 con 0x7f4e6c02fe60 2023-11-30T21:43:54.371 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-11-30T21:43:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.371+0000 7f4e90aea700 1 -- 172.21.15.27:0/984783076 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f4e6c02fe60 msgr2=0x7f4e6c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.371+0000 7f4e90aea700 1 --2- 172.21.15.27:0/984783076 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f4e6c02fe60 0x7f4e6c032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f4e7c0096a0 tx=0x7f4e7c006b40).stop 2023-11-30T21:43:54.374 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.371+0000 7f4e90aea700 1 -- 172.21.15.27:0/984783076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e8c0fab80 msgr2=0x7f4e8c10aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:54.374 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.371+0000 7f4e90aea700 1 --2- 172.21.15.27:0/984783076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e8c0fab80 0x7f4e8c10aa10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f4e7400e040 tx=0x7f4e74006ee0).stop 2023-11-30T21:43:54.374 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.371+0000 7f4e90aea700 1 -- 172.21.15.27:0/984783076 shutdown_connections 2023-11-30T21:43:54.374 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.371+0000 7f4e90aea700 1 --2- 172.21.15.27:0/984783076 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f4e6c02fe60 0x7f4e6c032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:54.374 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.371+0000 7f4e90aea700 1 --2- 172.21.15.27:0/984783076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e8c0fab80 0x7f4e8c10aa10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:54.375 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.371+0000 7f4e90aea700 1 -- 172.21.15.27:0/984783076 >> 172.21.15.27:0/984783076 conn(0x7f4e8c0f63c0 msgr2=0x7f4e8c070010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:54.375 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.371+0000 7f4e90aea700 1 -- 172.21.15.27:0/984783076 shutdown_connections 2023-11-30T21:43:54.375 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:54.371+0000 7f4e90aea700 1 -- 172.21.15.27:0/984783076 wait complete. 2023-11-30T21:43:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-11-30T21:43:55.078 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.075+0000 7f7eb9cb2700 1 Processor -- start 2023-11-30T21:43:55.078 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.075+0000 7f7eb9cb2700 1 -- start start 2023-11-30T21:43:55.078 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.075+0000 7f7eb9cb2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb40fd340 0x7f7eb40fd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:55.078 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.075+0000 7f7eb9cb2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb40fdc50 con 0x7f7eb40fd340 2023-11-30T21:43:55.078 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.076+0000 7f7eb37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb40fd340 0x7f7eb40fd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:55.078 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.076+0000 7f7eb37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb40fd340 0x7f7eb40fd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36708/0 (socket says 172.21.15.27:36708) 2023-11-30T21:43:55.078 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.076+0000 7f7eb37fe700 1 -- 172.21.15.27:0/3194789690 learned_addr learned my addr 172.21.15.27:0/3194789690 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:55.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.076+0000 7f7eb37fe700 1 -- 172.21.15.27:0/3194789690 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb40fdd90 con 0x7f7eb40fd340 2023-11-30T21:43:55.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.076+0000 7f7eb37fe700 1 --2- 172.21.15.27:0/3194789690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb40fd340 0x7f7eb40fd760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f7ea400b5f0 tx=0x7f7ea4006ee0).ready entity=mon.0 client_cookie=d3555ee0c63df94e server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:55.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.076+0000 7f7eb27fc700 1 -- 172.21.15.27:0/3194789690 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ea400d600 con 0x7f7eb40fd340 2023-11-30T21:43:55.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.076+0000 7f7eb27fc700 1 -- 172.21.15.27:0/3194789690 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7ea400dbd0 con 0x7f7eb40fd340 2023-11-30T21:43:55.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.076+0000 7f7eb27fc700 1 -- 172.21.15.27:0/3194789690 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ea4011c00 con 0x7f7eb40fd340 2023-11-30T21:43:55.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.077+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3194789690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb40fd340 msgr2=0x7f7eb40fd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:55.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.077+0000 7f7eb9cb2700 1 --2- 172.21.15.27:0/3194789690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb40fd340 0x7f7eb40fd760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f7ea400b5f0 tx=0x7f7ea4006ee0).stop 2023-11-30T21:43:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.077+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3194789690 shutdown_connections 2023-11-30T21:43:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.077+0000 7f7eb9cb2700 1 --2- 172.21.15.27:0/3194789690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb40fd340 0x7f7eb40fd760 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.077+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3194789690 >> 172.21.15.27:0/3194789690 conn(0x7f7eb40f5b40 msgr2=0x7f7eb40f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.077+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3194789690 shutdown_connections 2023-11-30T21:43:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.077+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3194789690 wait complete. 2023-11-30T21:43:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.078+0000 7f7eb9cb2700 1 Processor -- start 2023-11-30T21:43:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.078+0000 7f7eb9cb2700 1 -- start start 2023-11-30T21:43:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.078+0000 7f7eb9cb2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb40fd340 0x7f7eb410aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.078+0000 7f7eb9cb2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb410af30 con 0x7f7eb40fd340 2023-11-30T21:43:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.079+0000 7f7eb37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb40fd340 0x7f7eb410aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.079+0000 7f7eb37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb40fd340 0x7f7eb410aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36710/0 (socket says 172.21.15.27:36710) 2023-11-30T21:43:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.079+0000 7f7eb37fe700 1 -- 172.21.15.27:0/3749853873 learned_addr learned my addr 172.21.15.27:0/3749853873 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:55.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.079+0000 7f7eb37fe700 1 -- 172.21.15.27:0/3749853873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea400b200 con 0x7f7eb40fd340 2023-11-30T21:43:55.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.079+0000 7f7eb37fe700 1 --2- 172.21.15.27:0/3749853873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb40fd340 0x7f7eb410aa40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f7ea4002f40 tx=0x7f7ea4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:55.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.079+0000 7f7eabfff700 1 -- 172.21.15.27:0/3749853873 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ea401b410 con 0x7f7eb40fd340 2023-11-30T21:43:55.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.079+0000 7f7eabfff700 1 -- 172.21.15.27:0/3749853873 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7ea401b9e0 con 0x7f7eb40fd340 2023-11-30T21:43:55.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.079+0000 7f7eabfff700 1 -- 172.21.15.27:0/3749853873 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ea401a8b0 con 0x7f7eb40fd340 2023-11-30T21:43:55.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.079+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3749853873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb410b130 con 0x7f7eb40fd340 2023-11-30T21:43:55.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.079+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3749853873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb410b540 con 0x7f7eb40fd340 2023-11-30T21:43:55.084 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.080+0000 7f7eabfff700 1 -- 172.21.15.27:0/3749853873 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f7ea400f070 con 0x7f7eb40fd340 2023-11-30T21:43:55.084 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.080+0000 7f7eabfff700 1 --2- 172.21.15.27:0/3749853873 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f7e9402fe60 0x7f7e94032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:55.084 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.080+0000 7f7eabfff700 1 -- 172.21.15.27:0/3749853873 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7ea401bb50 con 0x7f7eb40fd340 2023-11-30T21:43:55.084 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.080+0000 7f7eb2ffd700 1 --2- 172.21.15.27:0/3749853873 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f7e9402fe60 0x7f7e94032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:55.084 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.081+0000 7f7eb2ffd700 1 --2- 172.21.15.27:0/3749853873 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f7e9402fe60 0x7f7e94032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f7e9c0096a0 tx=0x7f7e9c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:55.084 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.081+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3749853873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb4059100 con 0x7f7eb40fd340 2023-11-30T21:43:55.086 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.085+0000 7f7eabfff700 1 -- 172.21.15.27:0/3749853873 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ea4016070 con 0x7f7eb40fd340 2023-11-30T21:43:55.244 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.243+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3749853873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f7eb4104530 con 0x7f7eb40fd340 2023-11-30T21:43:55.247 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.246+0000 7f7eabfff700 1 -- 172.21.15.27:0/3749853873 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f7ea4022e80 con 0x7f7eb40fd340 2023-11-30T21:43:55.250 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.248+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3749853873 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f7e9402fe60 msgr2=0x7f7e94032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:55.250 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.248+0000 7f7eb9cb2700 1 --2- 172.21.15.27:0/3749853873 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f7e9402fe60 0x7f7e94032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f7e9c0096a0 tx=0x7f7e9c006b40).stop 2023-11-30T21:43:55.250 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.248+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3749853873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb40fd340 msgr2=0x7f7eb410aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:55.250 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.248+0000 7f7eb9cb2700 1 --2- 172.21.15.27:0/3749853873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb40fd340 0x7f7eb410aa40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f7ea4002f40 tx=0x7f7ea4006ee0).stop 2023-11-30T21:43:55.250 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.248+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3749853873 shutdown_connections 2023-11-30T21:43:55.250 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.248+0000 7f7eb9cb2700 1 --2- 172.21.15.27:0/3749853873 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f7e9402fe60 0x7f7e94032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:55.251 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.248+0000 7f7eb9cb2700 1 --2- 172.21.15.27:0/3749853873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7eb40fd340 0x7f7eb410aa40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:55.251 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.248+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3749853873 >> 172.21.15.27:0/3749853873 conn(0x7f7eb40f5b40 msgr2=0x7f7eb40ff2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:55.251 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.248+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3749853873 shutdown_connections 2023-11-30T21:43:55.251 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.248+0000 7f7eb9cb2700 1 -- 172.21.15.27:0/3749853873 wait complete. 2023-11-30T21:43:55.636 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-11-30T21:43:55.639 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:53.424085+0000 mgr.smithi027.brrjcl (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-11-30T21:43:55.639 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:43:53.424719+0000 mgr.smithi027.brrjcl (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-11-30T21:43:55.639 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:54.367378+0000 mgr.smithi027.brrjcl (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-11-30T21:43:55.639 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:43:54.367908+0000 mgr.smithi027.brrjcl (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-11-30T21:43:55.639 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:54.370185+0000 mon.smithi027 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.27:0/1800026764' entity='mgr.smithi027.brrjcl' 2023-11-30T21:43:55.639 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:55.246842+0000 mon.smithi027 (mon.0) 81 : audit [INF] from='client.? 172.21.15.27:0/3749853873' entity='client.admin' 2023-11-30T21:43:55.977 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.974+0000 7f8b8aa0d700 1 Processor -- start 2023-11-30T21:43:55.977 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.975+0000 7f8b8aa0d700 1 -- start start 2023-11-30T21:43:55.977 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.975+0000 7f8b8aa0d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b84070390 0x7f8b84100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:55.977 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.975+0000 7f8b8aa0d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b840fc5c0 con 0x7f8b84070390 2023-11-30T21:43:55.978 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.975+0000 7f8b89a0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b84070390 0x7f8b84100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:55.978 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.975+0000 7f8b89a0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b84070390 0x7f8b84100bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36722/0 (socket says 172.21.15.27:36722) 2023-11-30T21:43:55.978 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.975+0000 7f8b89a0b700 1 -- 172.21.15.27:0/3687897903 learned_addr learned my addr 172.21.15.27:0/3687897903 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:55.978 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.975+0000 7f8b89a0b700 1 -- 172.21.15.27:0/3687897903 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b841010c0 con 0x7f8b84070390 2023-11-30T21:43:55.978 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.976+0000 7f8b89a0b700 1 --2- 172.21.15.27:0/3687897903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b84070390 0x7f8b84100bd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f8b8000b5f0 tx=0x7f8b80006ee0).ready entity=mon.0 client_cookie=71163a9190671ecd server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:55.978 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.976+0000 7f8b88a09700 1 -- 172.21.15.27:0/3687897903 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b8000d600 con 0x7f8b84070390 2023-11-30T21:43:55.978 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.976+0000 7f8b88a09700 1 -- 172.21.15.27:0/3687897903 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8b8000dbd0 con 0x7f8b84070390 2023-11-30T21:43:55.978 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.976+0000 7f8b88a09700 1 -- 172.21.15.27:0/3687897903 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b80011c00 con 0x7f8b84070390 2023-11-30T21:43:55.978 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.976+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/3687897903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b84070390 msgr2=0x7f8b84100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:55.978 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.976+0000 7f8b8aa0d700 1 --2- 172.21.15.27:0/3687897903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b84070390 0x7f8b84100bd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f8b8000b5f0 tx=0x7f8b80006ee0).stop 2023-11-30T21:43:55.981 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.976+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/3687897903 shutdown_connections 2023-11-30T21:43:55.981 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.976+0000 7f8b8aa0d700 1 --2- 172.21.15.27:0/3687897903 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b84070390 0x7f8b84100bd0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:55.981 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.976+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/3687897903 >> 172.21.15.27:0/3687897903 conn(0x7f8b840f6210 msgr2=0x7f8b840f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:55.981 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.976+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/3687897903 shutdown_connections 2023-11-30T21:43:55.981 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.976+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/3687897903 wait complete. 2023-11-30T21:43:55.981 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.977+0000 7f8b8aa0d700 1 Processor -- start 2023-11-30T21:43:55.981 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.977+0000 7f8b8aa0d700 1 -- start start 2023-11-30T21:43:55.981 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.977+0000 7f8b8aa0d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b84070390 0x7f8b8410ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:55.981 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.977+0000 7f8b8aa0d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8410cf10 con 0x7f8b84070390 2023-11-30T21:43:55.982 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.977+0000 7f8b89a0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b84070390 0x7f8b8410ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:55.982 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.977+0000 7f8b89a0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b84070390 0x7f8b8410ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36732/0 (socket says 172.21.15.27:36732) 2023-11-30T21:43:55.982 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.977+0000 7f8b89a0b700 1 -- 172.21.15.27:0/4191644191 learned_addr learned my addr 172.21.15.27:0/4191644191 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:55.982 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.978+0000 7f8b89a0b700 1 -- 172.21.15.27:0/4191644191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8000b200 con 0x7f8b84070390 2023-11-30T21:43:55.982 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.978+0000 7f8b89a0b700 1 --2- 172.21.15.27:0/4191644191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b84070390 0x7f8b8410ca20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8b8000b190 tx=0x7f8b80006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:55.982 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.978+0000 7f8b727fc700 1 -- 172.21.15.27:0/4191644191 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b8001b410 con 0x7f8b84070390 2023-11-30T21:43:55.982 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.978+0000 7f8b727fc700 1 -- 172.21.15.27:0/4191644191 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8b8001b9e0 con 0x7f8b84070390 2023-11-30T21:43:55.982 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.978+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/4191644191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8410d110 con 0x7f8b84070390 2023-11-30T21:43:55.982 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.978+0000 7f8b727fc700 1 -- 172.21.15.27:0/4191644191 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b8001a880 con 0x7f8b84070390 2023-11-30T21:43:55.982 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.978+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/4191644191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8410d520 con 0x7f8b84070390 2023-11-30T21:43:55.982 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.979+0000 7f8b727fc700 1 -- 172.21.15.27:0/4191644191 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f8b8000f070 con 0x7f8b84070390 2023-11-30T21:43:55.982 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.979+0000 7f8b727fc700 1 --2- 172.21.15.27:0/4191644191 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8b6c02fe60 0x7f8b6c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:55.983 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.979+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/4191644191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b84045ad0 con 0x7f8b84070390 2023-11-30T21:43:55.983 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.979+0000 7f8b727fc700 1 -- 172.21.15.27:0/4191644191 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8b800413e0 con 0x7f8b84070390 2023-11-30T21:43:55.983 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.979+0000 7f8b8920a700 1 --2- 172.21.15.27:0/4191644191 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8b6c02fe60 0x7f8b6c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:55.983 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.979+0000 7f8b8920a700 1 --2- 172.21.15.27:0/4191644191 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8b6c02fe60 0x7f8b6c032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f8b740096a0 tx=0x7f8b74006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:55.984 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:55.984+0000 7f8b727fc700 1 -- 172.21.15.27:0/4191644191 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b80019e60 con 0x7f8b84070390 2023-11-30T21:43:56.142 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:56.141+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/4191644191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f8b84101bc0 con 0x7f8b84070390 2023-11-30T21:43:56.145 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:56.144+0000 7f8b727fc700 1 -- 172.21.15.27:0/4191644191 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f8b80016030 con 0x7f8b84070390 2023-11-30T21:43:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:56.146+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/4191644191 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8b6c02fe60 msgr2=0x7f8b6c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:56.146+0000 7f8b8aa0d700 1 --2- 172.21.15.27:0/4191644191 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8b6c02fe60 0x7f8b6c032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f8b740096a0 tx=0x7f8b74006b40).stop 2023-11-30T21:43:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:56.146+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/4191644191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b84070390 msgr2=0x7f8b8410ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:56.147 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:56.146+0000 7f8b8aa0d700 1 --2- 172.21.15.27:0/4191644191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b84070390 0x7f8b8410ca20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f8b8000b190 tx=0x7f8b80006ee0).stop 2023-11-30T21:43:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:56.146+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/4191644191 shutdown_connections 2023-11-30T21:43:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:56.146+0000 7f8b8aa0d700 1 --2- 172.21.15.27:0/4191644191 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8b6c02fe60 0x7f8b6c032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:56.148 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:56.146+0000 7f8b8aa0d700 1 --2- 172.21.15.27:0/4191644191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b84070390 0x7f8b8410ca20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:56.149 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:56.146+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/4191644191 >> 172.21.15.27:0/4191644191 conn(0x7f8b840f6210 msgr2=0x7f8b840f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:56.149 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:56.146+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/4191644191 shutdown_connections 2023-11-30T21:43:56.149 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:56.146+0000 7f8b8aa0d700 1 -- 172.21.15.27:0/4191644191 wait complete. 2023-11-30T21:43:56.533 INFO:teuthology.orchestra.run.smithi027.stderr:Enabling the dashboard module... 2023-11-30T21:43:56.533 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-11-30T21:43:57.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.116+0000 7f0cee1fe700 1 Processor -- start 2023-11-30T21:43:57.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.116+0000 7f0cee1fe700 1 -- start start 2023-11-30T21:43:57.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.116+0000 7f0cee1fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ce8072470 0x7f0ce8070a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:57.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.117+0000 7f0cee1fe700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce8072890 con 0x7f0ce8072470 2023-11-30T21:43:57.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.117+0000 7f0ce77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ce8072470 0x7f0ce8070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:57.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.117+0000 7f0ce77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ce8072470 0x7f0ce8070a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36740/0 (socket says 172.21.15.27:36740) 2023-11-30T21:43:57.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.117+0000 7f0ce77fe700 1 -- 172.21.15.27:0/252542385 learned_addr learned my addr 172.21.15.27:0/252542385 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:57.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.117+0000 7f0ce77fe700 1 -- 172.21.15.27:0/252542385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce8070f00 con 0x7f0ce8072470 2023-11-30T21:43:57.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.117+0000 7f0ce77fe700 1 --2- 172.21.15.27:0/252542385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ce8072470 0x7f0ce8070a10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f0cdc00b5f0 tx=0x7f0cdc006ee0).ready entity=mon.0 client_cookie=5a1088cc7ec99398 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:57.120 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.117+0000 7f0ce67fc700 1 -- 172.21.15.27:0/252542385 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cdc00d600 con 0x7f0ce8072470 2023-11-30T21:43:57.120 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.117+0000 7f0ce67fc700 1 -- 172.21.15.27:0/252542385 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0cdc00dbd0 con 0x7f0ce8072470 2023-11-30T21:43:57.120 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.117+0000 7f0ce67fc700 1 -- 172.21.15.27:0/252542385 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cdc011c00 con 0x7f0ce8072470 2023-11-30T21:43:57.122 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.118+0000 7f0cee1fe700 1 -- 172.21.15.27:0/252542385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ce8072470 msgr2=0x7f0ce8070a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:57.122 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.118+0000 7f0cee1fe700 1 --2- 172.21.15.27:0/252542385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ce8072470 0x7f0ce8070a10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f0cdc00b5f0 tx=0x7f0cdc006ee0).stop 2023-11-30T21:43:57.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.118+0000 7f0cee1fe700 1 -- 172.21.15.27:0/252542385 shutdown_connections 2023-11-30T21:43:57.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.118+0000 7f0cee1fe700 1 --2- 172.21.15.27:0/252542385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ce8072470 0x7f0ce8070a10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:57.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.118+0000 7f0cee1fe700 1 -- 172.21.15.27:0/252542385 >> 172.21.15.27:0/252542385 conn(0x7f0ce80f74c0 msgr2=0x7f0ce80f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:57.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.118+0000 7f0cee1fe700 1 -- 172.21.15.27:0/252542385 shutdown_connections 2023-11-30T21:43:57.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.118+0000 7f0cee1fe700 1 -- 172.21.15.27:0/252542385 wait complete. 2023-11-30T21:43:57.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.119+0000 7f0cee1fe700 1 Processor -- start 2023-11-30T21:43:57.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.119+0000 7f0cee1fe700 1 -- start start 2023-11-30T21:43:57.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.119+0000 7f0cee1fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ce8072470 0x7f0ce810ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:57.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.119+0000 7f0cee1fe700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce810d1c0 con 0x7f0ce8072470 2023-11-30T21:43:57.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.119+0000 7f0ce77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ce8072470 0x7f0ce810ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:57.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.119+0000 7f0ce77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ce8072470 0x7f0ce810ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36746/0 (socket says 172.21.15.27:36746) 2023-11-30T21:43:57.123 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.119+0000 7f0ce77fe700 1 -- 172.21.15.27:0/4024746398 learned_addr learned my addr 172.21.15.27:0/4024746398 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:57.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.119+0000 7f0ce77fe700 1 -- 172.21.15.27:0/4024746398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc00b200 con 0x7f0ce8072470 2023-11-30T21:43:57.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.120+0000 7f0ce77fe700 1 --2- 172.21.15.27:0/4024746398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ce8072470 0x7f0ce810ccd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f0cdc011820 tx=0x7f0cdc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:57.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.120+0000 7f0ccffff700 1 -- 172.21.15.27:0/4024746398 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cdc01b410 con 0x7f0ce8072470 2023-11-30T21:43:57.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.120+0000 7f0ccffff700 1 -- 172.21.15.27:0/4024746398 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0cdc01b9e0 con 0x7f0ce8072470 2023-11-30T21:43:57.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.120+0000 7f0cee1fe700 1 -- 172.21.15.27:0/4024746398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce810d3c0 con 0x7f0ce8072470 2023-11-30T21:43:57.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.120+0000 7f0ccffff700 1 -- 172.21.15.27:0/4024746398 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cdc01a8b0 con 0x7f0ce8072470 2023-11-30T21:43:57.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.120+0000 7f0cee1fe700 1 -- 172.21.15.27:0/4024746398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce810d7d0 con 0x7f0ce8072470 2023-11-30T21:43:57.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.121+0000 7f0ccffff700 1 -- 172.21.15.27:0/4024746398 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f0cdc00f070 con 0x7f0ce8072470 2023-11-30T21:43:57.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.121+0000 7f0cee1fe700 1 -- 172.21.15.27:0/4024746398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce8045ad0 con 0x7f0ce8072470 2023-11-30T21:43:57.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.121+0000 7f0ccffff700 1 --2- 172.21.15.27:0/4024746398 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f0cd402feb0 0x7f0cd4032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:57.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.121+0000 7f0ccffff700 1 -- 172.21.15.27:0/4024746398 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0cdc041470 con 0x7f0ce8072470 2023-11-30T21:43:57.124 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.121+0000 7f0ce6ffd700 1 --2- 172.21.15.27:0/4024746398 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f0cd402feb0 0x7f0cd4032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:57.125 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.121+0000 7f0ce6ffd700 1 --2- 172.21.15.27:0/4024746398 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f0cd402feb0 0x7f0cd4032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f0cd80096a0 tx=0x7f0cd8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:57.127 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.125+0000 7f0ccffff700 1 -- 172.21.15.27:0/4024746398 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cdc022310 con 0x7f0ce8072470 2023-11-30T21:43:57.322 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:57.320+0000 7f0cee1fe700 1 -- 172.21.15.27:0/4024746398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f0ce8101e10 con 0x7f0ce8072470 2023-11-30T21:43:57.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:57 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:56.145143+0000 mon.smithi027 (mon.0) 82 : audit [INF] from='client.? 172.21.15.27:0/4191644191' entity='client.admin' 2023-11-30T21:43:58.151 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:58.149+0000 7f0ccffff700 1 -- 172.21.15.27:0/4024746398 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f0cdc01aa10 con 0x7f0ce8072470 2023-11-30T21:43:58.152 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:58.151+0000 7f0ccffff700 1 -- 172.21.15.27:0/4024746398 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v12) v1 ==== 88+0+0 (secure 0 0 0) 0x7f0cdc006a70 con 0x7f0ce8072470 2023-11-30T21:43:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:58.153+0000 7f0cee1fe700 1 -- 172.21.15.27:0/4024746398 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f0cd402feb0 msgr2=0x7f0cd4032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:58.153+0000 7f0cee1fe700 1 --2- 172.21.15.27:0/4024746398 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f0cd402feb0 0x7f0cd4032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f0cd80096a0 tx=0x7f0cd8006b40).stop 2023-11-30T21:43:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:58.153+0000 7f0cee1fe700 1 -- 172.21.15.27:0/4024746398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ce8072470 msgr2=0x7f0ce810ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:58.153+0000 7f0cee1fe700 1 --2- 172.21.15.27:0/4024746398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ce8072470 0x7f0ce810ccd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f0cdc011820 tx=0x7f0cdc006ee0).stop 2023-11-30T21:43:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:58.153+0000 7f0cee1fe700 1 -- 172.21.15.27:0/4024746398 shutdown_connections 2023-11-30T21:43:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:58.153+0000 7f0cee1fe700 1 --2- 172.21.15.27:0/4024746398 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f0cd402feb0 0x7f0cd4032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:58.153+0000 7f0cee1fe700 1 --2- 172.21.15.27:0/4024746398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ce8072470 0x7f0ce810ccd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:58.153+0000 7f0cee1fe700 1 -- 172.21.15.27:0/4024746398 >> 172.21.15.27:0/4024746398 conn(0x7f0ce80f74c0 msgr2=0x7f0ce80f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:58.153+0000 7f0cee1fe700 1 -- 172.21.15.27:0/4024746398 shutdown_connections 2023-11-30T21:43:58.156 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:58.153+0000 7f0cee1fe700 1 -- 172.21.15.27:0/4024746398 wait complete. 2023-11-30T21:43:58.401 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:58 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:57.321974+0000 mon.smithi027 (mon.0) 83 : audit [INF] from='client.? 172.21.15.27:0/4024746398' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-30T21:43:58.544 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-30T21:43:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.078+0000 7f1f7170b700 1 Processor -- start 2023-11-30T21:43:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.078+0000 7f1f7170b700 1 -- start start 2023-11-30T21:43:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.078+0000 7f1f7170b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f6c0fab80 0x7f1f6c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.078+0000 7f1f7170b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c0fb490 con 0x7f1f6c0fab80 2023-11-30T21:43:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.079+0000 7f1f6affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f6c0fab80 0x7f1f6c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.079+0000 7f1f6affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f6c0fab80 0x7f1f6c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36780/0 (socket says 172.21.15.27:36780) 2023-11-30T21:43:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.079+0000 7f1f6affd700 1 -- 172.21.15.27:0/2910837489 learned_addr learned my addr 172.21.15.27:0/2910837489 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:59.081 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.079+0000 7f1f6affd700 1 -- 172.21.15.27:0/2910837489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c0fb5d0 con 0x7f1f6c0fab80 2023-11-30T21:43:59.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.079+0000 7f1f6affd700 1 --2- 172.21.15.27:0/2910837489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f6c0fab80 0x7f1f6c0fafa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f1f5c00b870 tx=0x7f1f5c006f70).ready entity=mon.0 client_cookie=91f867b54ee23c3d server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:59.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.080+0000 7f1f69ffb700 1 -- 172.21.15.27:0/2910837489 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f5c00da20 con 0x7f1f6c0fab80 2023-11-30T21:43:59.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.080+0000 7f1f69ffb700 1 -- 172.21.15.27:0/2910837489 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1f5c010470 con 0x7f1f6c0fab80 2023-11-30T21:43:59.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.080+0000 7f1f69ffb700 1 -- 172.21.15.27:0/2910837489 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f5c00f5b0 con 0x7f1f6c0fab80 2023-11-30T21:43:59.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.080+0000 7f1f7170b700 1 -- 172.21.15.27:0/2910837489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f6c0fab80 msgr2=0x7f1f6c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:59.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.080+0000 7f1f7170b700 1 --2- 172.21.15.27:0/2910837489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f6c0fab80 0x7f1f6c0fafa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f1f5c00b870 tx=0x7f1f5c006f70).stop 2023-11-30T21:43:59.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.081+0000 7f1f7170b700 1 -- 172.21.15.27:0/2910837489 shutdown_connections 2023-11-30T21:43:59.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.081+0000 7f1f7170b700 1 --2- 172.21.15.27:0/2910837489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f6c0fab80 0x7f1f6c0fafa0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:59.083 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.081+0000 7f1f7170b700 1 -- 172.21.15.27:0/2910837489 >> 172.21.15.27:0/2910837489 conn(0x7f1f6c0f63c0 msgr2=0x7f1f6c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:59.084 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.081+0000 7f1f7170b700 1 -- 172.21.15.27:0/2910837489 shutdown_connections 2023-11-30T21:43:59.084 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.081+0000 7f1f7170b700 1 -- 172.21.15.27:0/2910837489 wait complete. 2023-11-30T21:43:59.084 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.081+0000 7f1f7170b700 1 Processor -- start 2023-11-30T21:43:59.084 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.081+0000 7f1f7170b700 1 -- start start 2023-11-30T21:43:59.084 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.082+0000 7f1f7170b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f6c0fab80 0x7f1f6c10cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:59.084 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.082+0000 7f1f7170b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c10d070 con 0x7f1f6c0fab80 2023-11-30T21:43:59.084 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.082+0000 7f1f6affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f6c0fab80 0x7f1f6c10cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:43:59.084 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.082+0000 7f1f6affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f6c0fab80 0x7f1f6c10cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36788/0 (socket says 172.21.15.27:36788) 2023-11-30T21:43:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.082+0000 7f1f6affd700 1 -- 172.21.15.27:0/2527960348 learned_addr learned my addr 172.21.15.27:0/2527960348 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:43:59.087 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.082+0000 7f1f6affd700 1 -- 172.21.15.27:0/2527960348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5c00b480 con 0x7f1f6c0fab80 2023-11-30T21:43:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.082+0000 7f1f6affd700 1 --2- 172.21.15.27:0/2527960348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f6c0fab80 0x7f1f6c10cb80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f1f5c006a40 tx=0x7f1f5c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:43:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.082+0000 7f1f537fe700 1 -- 172.21.15.27:0/2527960348 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f5c00de00 con 0x7f1f6c0fab80 2023-11-30T21:43:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.082+0000 7f1f7170b700 1 -- 172.21.15.27:0/2527960348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6c10d270 con 0x7f1f6c0fab80 2023-11-30T21:43:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.082+0000 7f1f537fe700 1 -- 172.21.15.27:0/2527960348 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1f5c00fef0 con 0x7f1f6c0fab80 2023-11-30T21:43:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.082+0000 7f1f537fe700 1 -- 172.21.15.27:0/2527960348 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f5c019e90 con 0x7f1f6c0fab80 2023-11-30T21:43:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.083+0000 7f1f7170b700 1 -- 172.21.15.27:0/2527960348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6c10d680 con 0x7f1f6c0fab80 2023-11-30T21:43:59.088 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.084+0000 7f1f537fe700 1 -- 172.21.15.27:0/2527960348 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f1f5c02a070 con 0x7f1f6c0fab80 2023-11-30T21:43:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.084+0000 7f1f537fe700 1 --2- 172.21.15.27:0/2527960348 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f1f5802feb0 0x7f1f58032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:43:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.084+0000 7f1f7170b700 1 -- 172.21.15.27:0/2527960348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6c045ad0 con 0x7f1f6c0fab80 2023-11-30T21:43:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.084+0000 7f1f537fe700 1 -- 172.21.15.27:0/2527960348 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f1f5c041c90 con 0x7f1f6c0fab80 2023-11-30T21:43:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.084+0000 7f1f6a7fc700 1 -- 172.21.15.27:0/2527960348 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f1f5802feb0 msgr2=0x7f1f58032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2420177321 2023-11-30T21:43:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.084+0000 7f1f6a7fc700 1 --2- 172.21.15.27:0/2527960348 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f1f5802feb0 0x7f1f58032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-30T21:43:59.089 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.088+0000 7f1f537fe700 1 -- 172.21.15.27:0/2527960348 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f5c019630 con 0x7f1f6c0fab80 2023-11-30T21:43:59.260 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:59 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:58.151955+0000 mon.smithi027 (mon.0) 84 : audit [INF] from='client.? 172.21.15.27:0/4024746398' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-30T21:43:59.260 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:43:59 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:43:58.152007+0000 mon.smithi027 (mon.0) 85 : cluster [DBG] mgrmap e12: smithi027.brrjcl(active, since 7s) 2023-11-30T21:43:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.285+0000 7f1f6a7fc700 1 -- 172.21.15.27:0/2527960348 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f1f5802feb0 msgr2=0x7f1f58032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2420177321 2023-11-30T21:43:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.285+0000 7f1f6a7fc700 1 --2- 172.21.15.27:0/2527960348 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f1f5802feb0 0x7f1f58032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-30T21:43:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.285+0000 7f1f7170b700 1 -- 172.21.15.27:0/2527960348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f1f6c10e730 con 0x7f1f6c0fab80 2023-11-30T21:43:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.285+0000 7f1f537fe700 1 -- 172.21.15.27:0/2527960348 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v12) v1 ==== 56+0+104 (secure 0 0 0) 0x7f1f5c017b50 con 0x7f1f6c0fab80 2023-11-30T21:43:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: { 2023-11-30T21:43:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "epoch": 12, 2023-11-30T21:43:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "available": true, 2023-11-30T21:43:59.288 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "active_name": "smithi027.brrjcl", 2023-11-30T21:43:59.288 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-30T21:43:59.288 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: } 2023-11-30T21:43:59.290 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.288+0000 7f1f7170b700 1 -- 172.21.15.27:0/2527960348 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f1f5802feb0 msgr2=0x7f1f58032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-30T21:43:59.290 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.288+0000 7f1f7170b700 1 --2- 172.21.15.27:0/2527960348 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f1f5802feb0 0x7f1f58032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-30T21:43:59.290 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.288+0000 7f1f7170b700 1 -- 172.21.15.27:0/2527960348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f6c0fab80 msgr2=0x7f1f6c10cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:43:59.290 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.288+0000 7f1f7170b700 1 --2- 172.21.15.27:0/2527960348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f6c0fab80 0x7f1f6c10cb80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f1f5c006a40 tx=0x7f1f5c006f70).stop 2023-11-30T21:43:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.288+0000 7f1f7170b700 1 -- 172.21.15.27:0/2527960348 shutdown_connections 2023-11-30T21:43:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.288+0000 7f1f7170b700 1 --2- 172.21.15.27:0/2527960348 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f1f5802feb0 0x7f1f58032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-30T21:43:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.288+0000 7f1f7170b700 1 --2- 172.21.15.27:0/2527960348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f6c0fab80 0x7f1f6c10cb80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:43:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.288+0000 7f1f7170b700 1 -- 172.21.15.27:0/2527960348 >> 172.21.15.27:0/2527960348 conn(0x7f1f6c0f63c0 msgr2=0x7f1f6c0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:43:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.288+0000 7f1f7170b700 1 -- 172.21.15.27:0/2527960348 shutdown_connections 2023-11-30T21:43:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:43:59.288+0000 7f1f7170b700 1 -- 172.21.15.27:0/2527960348 wait complete. 2023-11-30T21:43:59.620 INFO:teuthology.orchestra.run.smithi027.stderr:Waiting for the mgr to restart... 2023-11-30T21:43:59.620 INFO:teuthology.orchestra.run.smithi027.stderr:Waiting for mgr epoch 12... 2023-11-30T21:43:59.620 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-30T21:44:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.019+0000 7f8d88867700 1 Processor -- start 2023-11-30T21:44:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.019+0000 7f8d88867700 1 -- start start 2023-11-30T21:44:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.019+0000 7f8d88867700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d840fab80 0x7f8d840fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.019+0000 7f8d88867700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d840fb490 con 0x7f8d840fab80 2023-11-30T21:44:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.020+0000 7f8d8259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d840fab80 0x7f8d840fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.020+0000 7f8d8259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d840fab80 0x7f8d840fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36804/0 (socket says 172.21.15.27:36804) 2023-11-30T21:44:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.020+0000 7f8d8259c700 1 -- 172.21.15.27:0/3022748795 learned_addr learned my addr 172.21.15.27:0/3022748795 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:44:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.020+0000 7f8d8259c700 1 -- 172.21.15.27:0/3022748795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d840fb5d0 con 0x7f8d840fab80 2023-11-30T21:44:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.020+0000 7f8d8259c700 1 --2- 172.21.15.27:0/3022748795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d840fab80 0x7f8d840fafa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f8d7400b5f0 tx=0x7f8d74006ee0).ready entity=mon.0 client_cookie=ec1c4ae13a828c5d server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.020+0000 7f8d8159a700 1 -- 172.21.15.27:0/3022748795 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d7400d600 con 0x7f8d840fab80 2023-11-30T21:44:00.023 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.020+0000 7f8d8159a700 1 -- 172.21.15.27:0/3022748795 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8d7400dbd0 con 0x7f8d840fab80 2023-11-30T21:44:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.020+0000 7f8d8159a700 1 -- 172.21.15.27:0/3022748795 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d74011c00 con 0x7f8d840fab80 2023-11-30T21:44:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.021+0000 7f8d88867700 1 -- 172.21.15.27:0/3022748795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d840fab80 msgr2=0x7f8d840fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.021+0000 7f8d88867700 1 --2- 172.21.15.27:0/3022748795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d840fab80 0x7f8d840fafa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f8d7400b5f0 tx=0x7f8d74006ee0).stop 2023-11-30T21:44:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.021+0000 7f8d88867700 1 -- 172.21.15.27:0/3022748795 shutdown_connections 2023-11-30T21:44:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.021+0000 7f8d88867700 1 --2- 172.21.15.27:0/3022748795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d840fab80 0x7f8d840fafa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.021+0000 7f8d88867700 1 -- 172.21.15.27:0/3022748795 >> 172.21.15.27:0/3022748795 conn(0x7f8d840f63c0 msgr2=0x7f8d840f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.021+0000 7f8d88867700 1 -- 172.21.15.27:0/3022748795 shutdown_connections 2023-11-30T21:44:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.021+0000 7f8d88867700 1 -- 172.21.15.27:0/3022748795 wait complete. 2023-11-30T21:44:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.022+0000 7f8d88867700 1 Processor -- start 2023-11-30T21:44:00.024 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.022+0000 7f8d88867700 1 -- start start 2023-11-30T21:44:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.022+0000 7f8d88867700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d840fab80 0x7f8d84109900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.022+0000 7f8d88867700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d84109df0 con 0x7f8d840fab80 2023-11-30T21:44:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.022+0000 7f8d8259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d840fab80 0x7f8d84109900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.022+0000 7f8d8259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d840fab80 0x7f8d84109900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36818/0 (socket says 172.21.15.27:36818) 2023-11-30T21:44:00.026 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.022+0000 7f8d8259c700 1 -- 172.21.15.27:0/654578185 learned_addr learned my addr 172.21.15.27:0/654578185 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:44:00.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.022+0000 7f8d8259c700 1 -- 172.21.15.27:0/654578185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7400b200 con 0x7f8d840fab80 2023-11-30T21:44:00.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.023+0000 7f8d8259c700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d840fab80 0x7f8d84109900 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f8d7400e040 tx=0x7f8d74006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:00.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.023+0000 7f8d72ffd700 1 -- 172.21.15.27:0/654578185 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d74011e70 con 0x7f8d840fab80 2023-11-30T21:44:00.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.023+0000 7f8d88867700 1 -- 172.21.15.27:0/654578185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d84109ff0 con 0x7f8d840fab80 2023-11-30T21:44:00.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.023+0000 7f8d72ffd700 1 -- 172.21.15.27:0/654578185 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8d74022890 con 0x7f8d840fab80 2023-11-30T21:44:00.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.023+0000 7f8d72ffd700 1 -- 172.21.15.27:0/654578185 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d740186e0 con 0x7f8d840fab80 2023-11-30T21:44:00.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.023+0000 7f8d88867700 1 -- 172.21.15.27:0/654578185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8410a400 con 0x7f8d840fab80 2023-11-30T21:44:00.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.024+0000 7f8d72ffd700 1 -- 172.21.15.27:0/654578185 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f8d74029060 con 0x7f8d840fab80 2023-11-30T21:44:00.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.024+0000 7f8d72ffd700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8d6802feb0 0x7f8d68032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:00.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.024+0000 7f8d88867700 1 -- 172.21.15.27:0/654578185 --> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d84059100 con 0x7f8d6802feb0 2023-11-30T21:44:00.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.024+0000 7f8d72ffd700 1 -- 172.21.15.27:0/654578185 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f8d74042e10 con 0x7f8d840fab80 2023-11-30T21:44:00.027 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.024+0000 7f8d81d9b700 1 -- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8d6802feb0 msgr2=0x7f8d68032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2420177321 2023-11-30T21:44:00.028 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.024+0000 7f8d81d9b700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8d6802feb0 0x7f8d68032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-30T21:44:00.226 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.225+0000 7f8d81d9b700 1 -- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8d6802feb0 msgr2=0x7f8d68032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2420177321 2023-11-30T21:44:00.226 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.225+0000 7f8d81d9b700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8d6802feb0 0x7f8d68032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-30T21:44:00.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:00 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:43:59.286510+0000 mon.smithi027 (mon.0) 86 : audit [DBG] from='client.? 172.21.15.27:0/2527960348' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-30T21:44:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.626+0000 7f8d81d9b700 1 -- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8d6802feb0 msgr2=0x7f8d68032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2420177321 2023-11-30T21:44:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:00.626+0000 7f8d81d9b700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8d6802feb0 0x7f8d68032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-30T21:44:01.429 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:01.427+0000 7f8d81d9b700 1 -- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8d6802feb0 msgr2=0x7f8d68032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2420177321 2023-11-30T21:44:01.429 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:01.427+0000 7f8d81d9b700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8d6802feb0 0x7f8d68032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-30T21:44:03.030 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:03.028+0000 7f8d81d9b700 1 -- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8d6802feb0 msgr2=0x7f8d68032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2420177321 2023-11-30T21:44:03.031 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:03.028+0000 7f8d81d9b700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8d6802feb0 0x7f8d68032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-30T21:44:03.454 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:03.452+0000 7f8d72ffd700 1 -- 172.21.15.27:0/654578185 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mgrmap(e 13) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f8d7400f070 con 0x7f8d840fab80 2023-11-30T21:44:03.454 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:03.452+0000 7f8d72ffd700 1 -- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8d6802feb0 msgr2=0x7f8d68032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-30T21:44:03.454 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:03.452+0000 7f8d72ffd700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8d6802feb0 0x7f8d68032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-30T21:44:04.458 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.453+0000 7f8d72ffd700 1 -- 172.21.15.27:0/654578185 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f8d74010d60 con 0x7f8d840fab80 2023-11-30T21:44:04.459 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.454+0000 7f8d72ffd700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8d68033990 0x7f8d68035d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:04.459 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.454+0000 7f8d72ffd700 1 -- 172.21.15.27:0/654578185 --> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d84059100 con 0x7f8d68033990 2023-11-30T21:44:04.459 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.454+0000 7f8d81d9b700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8d68033990 0x7f8d68035d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:04.459 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.455+0000 7f8d81d9b700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8d68033990 0x7f8d68035d80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f8d78006fd0 tx=0x7f8d78006cf0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:04.459 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.455+0000 7f8d72ffd700 1 -- 172.21.15.27:0/654578185 <== mgr.14162 v2:172.21.15.27:6800/4041101911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f8d84059100 con 0x7f8d68033990 2023-11-30T21:44:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.462+0000 7f8d88867700 1 -- 172.21.15.27:0/654578185 --> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f8d84045ad0 con 0x7f8d68033990 2023-11-30T21:44:04.463 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.462+0000 7f8d72ffd700 1 -- 172.21.15.27:0/654578185 <== mgr.14162 v2:172.21.15.27:6800/4041101911 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f8d84045ad0 con 0x7f8d68033990 2023-11-30T21:44:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: { 2023-11-30T21:44:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "mgrmap_epoch": 14, 2023-11-30T21:44:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: "initialized": true 2023-11-30T21:44:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: } 2023-11-30T21:44:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.462+0000 7f8d88867700 1 -- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8d68033990 msgr2=0x7f8d68035d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.462+0000 7f8d88867700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8d68033990 0x7f8d68035d80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f8d78006fd0 tx=0x7f8d78006cf0).stop 2023-11-30T21:44:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.462+0000 7f8d88867700 1 -- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d840fab80 msgr2=0x7f8d84109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.462+0000 7f8d88867700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d840fab80 0x7f8d84109900 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f8d7400e040 tx=0x7f8d74006ee0).stop 2023-11-30T21:44:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.462+0000 7f8d88867700 1 -- 172.21.15.27:0/654578185 shutdown_connections 2023-11-30T21:44:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.462+0000 7f8d88867700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8d68033990 0x7f8d68035d80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:04.465 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.462+0000 7f8d88867700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d840fab80 0x7f8d84109900 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.463+0000 7f8d88867700 1 --2- 172.21.15.27:0/654578185 >> [v2:172.21.15.27:6800/2420177321,v1:172.21.15.27:6801/2420177321] conn(0x7f8d6802feb0 0x7f8d68032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-30T21:44:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.463+0000 7f8d88867700 1 -- 172.21.15.27:0/654578185 >> 172.21.15.27:0/654578185 conn(0x7f8d840f63c0 msgr2=0x7f8d840f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.463+0000 7f8d88867700 1 -- 172.21.15.27:0/654578185 shutdown_connections 2023-11-30T21:44:04.466 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:04.463+0000 7f8d88867700 1 -- 172.21.15.27:0/654578185 wait complete. 2023-11-30T21:44:04.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:03.398716+0000 mon.smithi027 (mon.0) 87 : cluster [INF] Active manager daemon smithi027.brrjcl restarted 2023-11-30T21:44:04.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:03.399184+0000 mon.smithi027 (mon.0) 88 : cluster [INF] Activating manager daemon smithi027.brrjcl 2023-11-30T21:44:04.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:03.403246+0000 mon.smithi027 (mon.0) 89 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-11-30T21:44:04.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:03.454342+0000 mon.smithi027 (mon.0) 90 : cluster [DBG] mgrmap e13: smithi027.brrjcl(active, starting, since 0.0552482s) 2023-11-30T21:44:04.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:03.455458+0000 mon.smithi027 (mon.0) 91 : audit [DBG] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2023-11-30T21:44:04.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:03.456104+0000 mon.smithi027 (mon.0) 92 : audit [DBG] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mgr metadata", "who": "smithi027.brrjcl", "id": "smithi027.brrjcl"}]: dispatch 2023-11-30T21:44:04.762 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:03.456967+0000 mon.smithi027 (mon.0) 93 : audit [DBG] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-30T21:44:04.762 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:03.457196+0000 mon.smithi027 (mon.0) 94 : audit [DBG] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-30T21:44:04.762 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:03.457396+0000 mon.smithi027 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-30T21:44:04.762 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:03.459459+0000 mon.smithi027 (mon.0) 96 : cluster [INF] Manager daemon smithi027.brrjcl is now available 2023-11-30T21:44:04.762 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:03.562889+0000 mon.smithi027 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-30T21:44:04.762 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:03.564426+0000 mon.smithi027 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-30T21:44:04.762 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:03.581221+0000 mon.smithi027 (mon.0) 99 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/mirror_snapshot_schedule"}]: dispatch 2023-11-30T21:44:04.762 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:03.583943+0000 mon.smithi027 (mon.0) 100 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/trash_purge_schedule"}]: dispatch 2023-11-30T21:44:04.768 INFO:teuthology.orchestra.run.smithi027.stderr:mgr epoch 12 is available 2023-11-30T21:44:04.768 INFO:teuthology.orchestra.run.smithi027.stderr:Generating a dashboard self-signed certificate... 2023-11-30T21:44:04.768 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-11-30T21:44:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.111+0000 7fe1b61b8700 1 Processor -- start 2023-11-30T21:44:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.111+0000 7fe1b61b8700 1 -- start start 2023-11-30T21:44:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.111+0000 7fe1b61b8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1b00fab90 0x7fe1b00fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.111+0000 7fe1b61b8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b00fb4a0 con 0x7fe1b00fab90 2023-11-30T21:44:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.111+0000 7fe1af7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1b00fab90 0x7fe1b00fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.111+0000 7fe1af7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1b00fab90 0x7fe1b00fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37316/0 (socket says 172.21.15.27:37316) 2023-11-30T21:44:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.111+0000 7fe1af7fe700 1 -- 172.21.15.27:0/1691428434 learned_addr learned my addr 172.21.15.27:0/1691428434 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:44:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.112+0000 7fe1af7fe700 1 -- 172.21.15.27:0/1691428434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b00fb5e0 con 0x7fe1b00fab90 2023-11-30T21:44:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.112+0000 7fe1af7fe700 1 --2- 172.21.15.27:0/1691428434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1b00fab90 0x7fe1b00fafb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fe1a000b5f0 tx=0x7fe1a0006ee0).ready entity=mon.0 client_cookie=e6a0a1cadd3d22bc server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.112+0000 7fe1ae7fc700 1 -- 172.21.15.27:0/1691428434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1a000d600 con 0x7fe1b00fab90 2023-11-30T21:44:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.112+0000 7fe1ae7fc700 1 -- 172.21.15.27:0/1691428434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe1a000dbd0 con 0x7fe1b00fab90 2023-11-30T21:44:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.112+0000 7fe1ae7fc700 1 -- 172.21.15.27:0/1691428434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1a0011c00 con 0x7fe1b00fab90 2023-11-30T21:44:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.113+0000 7fe1b61b8700 1 -- 172.21.15.27:0/1691428434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1b00fab90 msgr2=0x7fe1b00fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.113+0000 7fe1b61b8700 1 --2- 172.21.15.27:0/1691428434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1b00fab90 0x7fe1b00fafb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fe1a000b5f0 tx=0x7fe1a0006ee0).stop 2023-11-30T21:44:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.113+0000 7fe1b61b8700 1 -- 172.21.15.27:0/1691428434 shutdown_connections 2023-11-30T21:44:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.113+0000 7fe1b61b8700 1 --2- 172.21.15.27:0/1691428434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1b00fab90 0x7fe1b00fafb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.113+0000 7fe1b61b8700 1 -- 172.21.15.27:0/1691428434 >> 172.21.15.27:0/1691428434 conn(0x7fe1b00f63b0 msgr2=0x7fe1b00f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.113+0000 7fe1b61b8700 1 -- 172.21.15.27:0/1691428434 shutdown_connections 2023-11-30T21:44:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.113+0000 7fe1b61b8700 1 -- 172.21.15.27:0/1691428434 wait complete. 2023-11-30T21:44:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.113+0000 7fe1b61b8700 1 Processor -- start 2023-11-30T21:44:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.113+0000 7fe1b61b8700 1 -- start start 2023-11-30T21:44:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.114+0000 7fe1b61b8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1b00fab90 0x7fe1b0100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.114+0000 7fe1b61b8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b00ff2a0 con 0x7fe1b00fab90 2023-11-30T21:44:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.114+0000 7fe1af7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1b00fab90 0x7fe1b0100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.114+0000 7fe1af7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1b00fab90 0x7fe1b0100e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37324/0 (socket says 172.21.15.27:37324) 2023-11-30T21:44:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.114+0000 7fe1af7fe700 1 -- 172.21.15.27:0/2762431209 learned_addr learned my addr 172.21.15.27:0/2762431209 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:44:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.114+0000 7fe1af7fe700 1 -- 172.21.15.27:0/2762431209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a000b200 con 0x7fe1b00fab90 2023-11-30T21:44:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.114+0000 7fe1af7fe700 1 --2- 172.21.15.27:0/2762431209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1b00fab90 0x7fe1b0100e20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fe1a000b1d0 tx=0x7fe1a0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.114+0000 7fe197fff700 1 -- 172.21.15.27:0/2762431209 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1a001b410 con 0x7fe1b00fab90 2023-11-30T21:44:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.114+0000 7fe197fff700 1 -- 172.21.15.27:0/2762431209 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe1a001b9e0 con 0x7fe1b00fab90 2023-11-30T21:44:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.114+0000 7fe1b61b8700 1 -- 172.21.15.27:0/2762431209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b00ff440 con 0x7fe1b00fab90 2023-11-30T21:44:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.114+0000 7fe197fff700 1 -- 172.21.15.27:0/2762431209 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1a001a840 con 0x7fe1b00fab90 2023-11-30T21:44:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.114+0000 7fe1b61b8700 1 -- 172.21.15.27:0/2762431209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b00ff850 con 0x7fe1b00fab90 2023-11-30T21:44:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.115+0000 7fe195ffb700 1 -- 172.21.15.27:0/2762431209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe188004fa0 con 0x7fe1b00fab90 2023-11-30T21:44:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.116+0000 7fe197fff700 1 -- 172.21.15.27:0/2762431209 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fe1a000f070 con 0x7fe1b00fab90 2023-11-30T21:44:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.116+0000 7fe197fff700 1 --2- 172.21.15.27:0/2762431209 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe19c02fda0 0x7fe19c032260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.116+0000 7fe197fff700 1 -- 172.21.15.27:0/2762431209 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe1a0042090 con 0x7fe1b00fab90 2023-11-30T21:44:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.116+0000 7fe1aeffd700 1 --2- 172.21.15.27:0/2762431209 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe19c02fda0 0x7fe19c032260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.117+0000 7fe1aeffd700 1 --2- 172.21.15.27:0/2762431209 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe19c02fda0 0x7fe19c032260 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe1a40096a0 tx=0x7fe1a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:05.121 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.120+0000 7fe197fff700 1 -- 172.21.15.27:0/2762431209 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1a002db70 con 0x7fe1b00fab90 2023-11-30T21:44:05.285 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.284+0000 7fe195ffb700 1 -- 172.21.15.27:0/2762431209 --> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fe188000bc0 con 0x7fe19c02fda0 2023-11-30T21:44:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.413+0000 7fe197fff700 1 -- 172.21.15.27:0/2762431209 <== mgr.14162 v2:172.21.15.27:6800/4041101911 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fe188000bc0 con 0x7fe19c02fda0 2023-11-30T21:44:05.415 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: Self-signed certificate created 2023-11-30T21:44:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.415+0000 7fe195ffb700 1 -- 172.21.15.27:0/2762431209 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe19c02fda0 msgr2=0x7fe19c032260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.415+0000 7fe195ffb700 1 --2- 172.21.15.27:0/2762431209 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe19c02fda0 0x7fe19c032260 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe1a40096a0 tx=0x7fe1a4006b40).stop 2023-11-30T21:44:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.415+0000 7fe195ffb700 1 -- 172.21.15.27:0/2762431209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1b00fab90 msgr2=0x7fe1b0100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.415+0000 7fe195ffb700 1 --2- 172.21.15.27:0/2762431209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1b00fab90 0x7fe1b0100e20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fe1a000b1d0 tx=0x7fe1a0006ee0).stop 2023-11-30T21:44:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.415+0000 7fe195ffb700 1 -- 172.21.15.27:0/2762431209 shutdown_connections 2023-11-30T21:44:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.415+0000 7fe195ffb700 1 --2- 172.21.15.27:0/2762431209 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe19c02fda0 0x7fe19c032260 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:05.417 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.415+0000 7fe195ffb700 1 --2- 172.21.15.27:0/2762431209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1b00fab90 0x7fe1b0100e20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:05.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.415+0000 7fe195ffb700 1 -- 172.21.15.27:0/2762431209 >> 172.21.15.27:0/2762431209 conn(0x7fe1b00f63b0 msgr2=0x7fe1b00700c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:05.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.415+0000 7fe195ffb700 1 -- 172.21.15.27:0/2762431209 shutdown_connections 2023-11-30T21:44:05.418 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:05.415+0000 7fe195ffb700 1 -- 172.21.15.27:0/2762431209 wait complete. 2023-11-30T21:44:05.760 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:04.456423+0000 mgr.smithi027.brrjcl (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-30T21:44:05.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:04.456639+0000 mon.smithi027 (mon.0) 101 : cluster [DBG] mgrmap e14: smithi027.brrjcl(active, since 1.05754s) 2023-11-30T21:44:05.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:04.463329+0000 mgr.smithi027.brrjcl (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-30T21:44:05.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:05.404932+0000 mon.smithi027 (mon.0) 102 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:44:05.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:05.414446+0000 mon.smithi027 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:44:05.918 INFO:teuthology.orchestra.run.smithi027.stderr:Creating initial admin user... 2023-11-30T21:44:05.918 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpfn1k4x68:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-11-30T21:44:06.260 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.257+0000 7f9936ebc700 1 Processor -- start 2023-11-30T21:44:06.260 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.258+0000 7f9936ebc700 1 -- start start 2023-11-30T21:44:06.260 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.258+0000 7f9936ebc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9930070430 0x7f99300fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.258+0000 7f9936ebc700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99300fec20 con 0x7f9930070430 2023-11-30T21:44:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.258+0000 7f9935eba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9930070430 0x7f99300fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.258+0000 7f9935eba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9930070430 0x7f99300fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37334/0 (socket says 172.21.15.27:37334) 2023-11-30T21:44:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.258+0000 7f9935eba700 1 -- 172.21.15.27:0/2236707316 learned_addr learned my addr 172.21.15.27:0/2236707316 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:44:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.258+0000 7f9935eba700 1 -- 172.21.15.27:0/2236707316 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99300fed60 con 0x7f9930070430 2023-11-30T21:44:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.258+0000 7f9935eba700 1 --2- 172.21.15.27:0/2236707316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9930070430 0x7f99300fe730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f992c00b5f0 tx=0x7f992c006ee0).ready entity=mon.0 client_cookie=70890b33a85a5310 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.259+0000 7f9934eb8700 1 -- 172.21.15.27:0/2236707316 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f992c00d600 con 0x7f9930070430 2023-11-30T21:44:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.259+0000 7f9934eb8700 1 -- 172.21.15.27:0/2236707316 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f992c00dbd0 con 0x7f9930070430 2023-11-30T21:44:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.259+0000 7f9934eb8700 1 -- 172.21.15.27:0/2236707316 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f992c011c00 con 0x7f9930070430 2023-11-30T21:44:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.259+0000 7f9936ebc700 1 -- 172.21.15.27:0/2236707316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9930070430 msgr2=0x7f99300fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:06.261 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.259+0000 7f9936ebc700 1 --2- 172.21.15.27:0/2236707316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9930070430 0x7f99300fe730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f992c00b5f0 tx=0x7f992c006ee0).stop 2023-11-30T21:44:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.259+0000 7f9936ebc700 1 -- 172.21.15.27:0/2236707316 shutdown_connections 2023-11-30T21:44:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.259+0000 7f9936ebc700 1 --2- 172.21.15.27:0/2236707316 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9930070430 0x7f99300fe730 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.259+0000 7f9936ebc700 1 -- 172.21.15.27:0/2236707316 >> 172.21.15.27:0/2236707316 conn(0x7f99300f6210 msgr2=0x7f99300f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.259+0000 7f9936ebc700 1 -- 172.21.15.27:0/2236707316 shutdown_connections 2023-11-30T21:44:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.260+0000 7f9936ebc700 1 -- 172.21.15.27:0/2236707316 wait complete. 2023-11-30T21:44:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.260+0000 7f9936ebc700 1 Processor -- start 2023-11-30T21:44:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.260+0000 7f9936ebc700 1 -- start start 2023-11-30T21:44:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.260+0000 7f9936ebc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9930070430 0x7f99300fc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.260+0000 7f9936ebc700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99300fa630 con 0x7f9930070430 2023-11-30T21:44:06.263 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.260+0000 7f9935eba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9930070430 0x7f99300fc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.260+0000 7f9935eba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9930070430 0x7f99300fc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37346/0 (socket says 172.21.15.27:37346) 2023-11-30T21:44:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.260+0000 7f9935eba700 1 -- 172.21.15.27:0/1253472229 learned_addr learned my addr 172.21.15.27:0/1253472229 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:44:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.261+0000 7f9935eba700 1 -- 172.21.15.27:0/1253472229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992c00b200 con 0x7f9930070430 2023-11-30T21:44:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.261+0000 7f9935eba700 1 --2- 172.21.15.27:0/1253472229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9930070430 0x7f99300fc1b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f992c00b190 tx=0x7f992c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.261+0000 7f991e7fc700 1 -- 172.21.15.27:0/1253472229 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f992c01b410 con 0x7f9930070430 2023-11-30T21:44:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.261+0000 7f991e7fc700 1 -- 172.21.15.27:0/1253472229 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f992c01b9e0 con 0x7f9930070430 2023-11-30T21:44:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.261+0000 7f9936ebc700 1 -- 172.21.15.27:0/1253472229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99300fa7d0 con 0x7f9930070430 2023-11-30T21:44:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.261+0000 7f991e7fc700 1 -- 172.21.15.27:0/1253472229 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f992c01a880 con 0x7f9930070430 2023-11-30T21:44:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.261+0000 7f9936ebc700 1 -- 172.21.15.27:0/1253472229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99300fabe0 con 0x7f9930070430 2023-11-30T21:44:06.264 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.262+0000 7f991e7fc700 1 -- 172.21.15.27:0/1253472229 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f992c00f070 con 0x7f9930070430 2023-11-30T21:44:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.262+0000 7f991e7fc700 1 --2- 172.21.15.27:0/1253472229 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f991802fd50 0x7f9918032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.262+0000 7f991e7fc700 1 -- 172.21.15.27:0/1253472229 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f992c041460 con 0x7f9930070430 2023-11-30T21:44:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.262+0000 7f9936ebc700 1 -- 172.21.15.27:0/1253472229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9930045ad0 con 0x7f9930070430 2023-11-30T21:44:06.265 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.262+0000 7f99356b9700 1 --2- 172.21.15.27:0/1253472229 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f991802fd50 0x7f9918032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:06.266 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.263+0000 7f99356b9700 1 --2- 172.21.15.27:0/1253472229 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f991802fd50 0x7f9918032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f99200096a0 tx=0x7f9920006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:06.271 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.267+0000 7f991e7fc700 1 -- 172.21.15.27:0/1253472229 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f992c016070 con 0x7f9930070430 2023-11-30T21:44:06.406 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.404+0000 7f991e7fc700 1 -- 172.21.15.27:0/1253472229 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f992c01bb50 con 0x7f9930070430 2023-11-30T21:44:06.441 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.440+0000 7f9936ebc700 1 -- 172.21.15.27:0/1253472229 --> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9930058f20 con 0x7f991802fd50 2023-11-30T21:44:06.687 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.686+0000 7f991e7fc700 1 -- 172.21.15.27:0/1253472229 <== mgr.14162 v2:172.21.15.27:6800/4041101911 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f9930058f20 con 0x7f991802fd50 2023-11-30T21:44:06.687 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$YjyoMOvN.7E9KDuDVcrnvePkjW.HvBRe5geVq.2NgXcaO.tQNIcLi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1701380646, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-30T21:44:06.689 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.687+0000 7f9936ebc700 1 -- 172.21.15.27:0/1253472229 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f991802fd50 msgr2=0x7f9918032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:06.689 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.687+0000 7f9936ebc700 1 --2- 172.21.15.27:0/1253472229 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f991802fd50 0x7f9918032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f99200096a0 tx=0x7f9920006b40).stop 2023-11-30T21:44:06.689 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.687+0000 7f9936ebc700 1 -- 172.21.15.27:0/1253472229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9930070430 msgr2=0x7f99300fc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:06.690 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.687+0000 7f9936ebc700 1 --2- 172.21.15.27:0/1253472229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9930070430 0x7f99300fc1b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f992c00b190 tx=0x7f992c006ee0).stop 2023-11-30T21:44:06.690 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.687+0000 7f9936ebc700 1 -- 172.21.15.27:0/1253472229 shutdown_connections 2023-11-30T21:44:06.690 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.687+0000 7f9936ebc700 1 --2- 172.21.15.27:0/1253472229 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f991802fd50 0x7f9918032210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:06.690 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.687+0000 7f9936ebc700 1 --2- 172.21.15.27:0/1253472229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9930070430 0x7f99300fc1b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:06.690 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.687+0000 7f9936ebc700 1 -- 172.21.15.27:0/1253472229 >> 172.21.15.27:0/1253472229 conn(0x7f99300f6210 msgr2=0x7f99300f7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:06.690 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.688+0000 7f9936ebc700 1 -- 172.21.15.27:0/1253472229 shutdown_connections 2023-11-30T21:44:06.690 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:06.688+0000 7f9936ebc700 1 -- 172.21.15.27:0/1253472229 wait complete. 2023-11-30T21:44:06.708 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:06 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:05.285506+0000 mgr.smithi027.brrjcl (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-30T21:44:06.708 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:06 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:06.406418+0000 mon.smithi027 (mon.0) 104 : cluster [DBG] mgrmap e15: smithi027.brrjcl(active, since 3s) 2023-11-30T21:44:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:Fetching dashboard port number... 2023-11-30T21:44:07.177 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp3c2eh1oc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpce8al19t:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-11-30T21:44:07.553 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.551+0000 7f91685bb700 1 Processor -- start 2023-11-30T21:44:07.553 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.551+0000 7f91685bb700 1 -- start start 2023-11-30T21:44:07.553 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.551+0000 7f91685bb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9160070430 0x7f91600fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:07.553 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.551+0000 7f91685bb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91600fedd0 con 0x7f9160070430 2023-11-30T21:44:07.553 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.551+0000 7f9166357700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9160070430 0x7f91600fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:07.559 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.552+0000 7f9166357700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9160070430 0x7f91600fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37350/0 (socket says 172.21.15.27:37350) 2023-11-30T21:44:07.559 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.552+0000 7f9166357700 1 -- 172.21.15.27:0/4226084025 learned_addr learned my addr 172.21.15.27:0/4226084025 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:44:07.559 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.552+0000 7f9166357700 1 -- 172.21.15.27:0/4226084025 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91600fef10 con 0x7f9160070430 2023-11-30T21:44:07.559 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.552+0000 7f9166357700 1 --2- 172.21.15.27:0/4226084025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9160070430 0x7f91600fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f915000b5f0 tx=0x7f9150006ee0).ready entity=mon.0 client_cookie=502c4c44d7d6b3c8 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:07.559 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.552+0000 7f9165355700 1 -- 172.21.15.27:0/4226084025 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f915000d600 con 0x7f9160070430 2023-11-30T21:44:07.559 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.552+0000 7f9165355700 1 -- 172.21.15.27:0/4226084025 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f915000dbd0 con 0x7f9160070430 2023-11-30T21:44:07.559 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.552+0000 7f9165355700 1 -- 172.21.15.27:0/4226084025 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9150011c00 con 0x7f9160070430 2023-11-30T21:44:07.559 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.553+0000 7f91685bb700 1 -- 172.21.15.27:0/4226084025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9160070430 msgr2=0x7f91600fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:07.559 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.553+0000 7f91685bb700 1 --2- 172.21.15.27:0/4226084025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9160070430 0x7f91600fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f915000b5f0 tx=0x7f9150006ee0).stop 2023-11-30T21:44:07.559 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.553+0000 7f91685bb700 1 -- 172.21.15.27:0/4226084025 shutdown_connections 2023-11-30T21:44:07.560 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.553+0000 7f91685bb700 1 --2- 172.21.15.27:0/4226084025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9160070430 0x7f91600fe8e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:07.560 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.553+0000 7f91685bb700 1 -- 172.21.15.27:0/4226084025 >> 172.21.15.27:0/4226084025 conn(0x7f91600f63c0 msgr2=0x7f91600f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:07.560 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.553+0000 7f91685bb700 1 -- 172.21.15.27:0/4226084025 shutdown_connections 2023-11-30T21:44:07.560 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.553+0000 7f91685bb700 1 -- 172.21.15.27:0/4226084025 wait complete. 2023-11-30T21:44:07.560 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.554+0000 7f91685bb700 1 Processor -- start 2023-11-30T21:44:07.560 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.554+0000 7f91685bb700 1 -- start start 2023-11-30T21:44:07.560 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.554+0000 7f91685bb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9160070430 0x7f91600fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:07.560 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.554+0000 7f91685bb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91600fa7e0 con 0x7f9160070430 2023-11-30T21:44:07.560 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.554+0000 7f9166357700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9160070430 0x7f91600fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:07.560 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.554+0000 7f9166357700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9160070430 0x7f91600fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37356/0 (socket says 172.21.15.27:37356) 2023-11-30T21:44:07.560 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.554+0000 7f9166357700 1 -- 172.21.15.27:0/3321459034 learned_addr learned my addr 172.21.15.27:0/3321459034 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:44:07.561 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.554+0000 7f9166357700 1 -- 172.21.15.27:0/3321459034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915000b200 con 0x7f9160070430 2023-11-30T21:44:07.561 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.554+0000 7f9166357700 1 --2- 172.21.15.27:0/3321459034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9160070430 0x7f91600fc360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f915000b190 tx=0x7f9150006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:07.561 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.555+0000 7f914effd700 1 -- 172.21.15.27:0/3321459034 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f915001b410 con 0x7f9160070430 2023-11-30T21:44:07.561 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.555+0000 7f914effd700 1 -- 172.21.15.27:0/3321459034 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f915001b9e0 con 0x7f9160070430 2023-11-30T21:44:07.562 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.555+0000 7f91685bb700 1 -- 172.21.15.27:0/3321459034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91600fa980 con 0x7f9160070430 2023-11-30T21:44:07.562 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.555+0000 7f914effd700 1 -- 172.21.15.27:0/3321459034 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f915001a880 con 0x7f9160070430 2023-11-30T21:44:07.562 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.555+0000 7f91685bb700 1 -- 172.21.15.27:0/3321459034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91600fad90 con 0x7f9160070430 2023-11-30T21:44:07.562 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.555+0000 7f914effd700 1 -- 172.21.15.27:0/3321459034 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f915000f070 con 0x7f9160070430 2023-11-30T21:44:07.562 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.556+0000 7f914effd700 1 --2- 172.21.15.27:0/3321459034 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f914802feb0 0x7f9148032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:07.562 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.556+0000 7f914effd700 1 -- 172.21.15.27:0/3321459034 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9150041740 con 0x7f9160070430 2023-11-30T21:44:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.556+0000 7f9165b56700 1 --2- 172.21.15.27:0/3321459034 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f914802feb0 0x7f9148032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.556+0000 7f91685bb700 1 -- 172.21.15.27:0/3321459034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9160059100 con 0x7f9160070430 2023-11-30T21:44:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.556+0000 7f9165b56700 1 --2- 172.21.15.27:0/3321459034 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f914802feb0 0x7f9148032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f915c0096a0 tx=0x7f915c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.560+0000 7f914effd700 1 -- 172.21.15.27:0/3321459034 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9150022d20 con 0x7f9160070430 2023-11-30T21:44:07.719 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.718+0000 7f91685bb700 1 -- 172.21.15.27:0/3321459034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f9160045ad0 con 0x7f9160070430 2023-11-30T21:44:07.719 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.718+0000 7f914effd700 1 -- 172.21.15.27:0/3321459034 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7f9150022310 con 0x7f9160070430 2023-11-30T21:44:07.719 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 8443 2023-11-30T21:44:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.720+0000 7f91685bb700 1 -- 172.21.15.27:0/3321459034 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f914802feb0 msgr2=0x7f9148032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.720+0000 7f91685bb700 1 --2- 172.21.15.27:0/3321459034 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f914802feb0 0x7f9148032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f915c0096a0 tx=0x7f915c006b40).stop 2023-11-30T21:44:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.720+0000 7f91685bb700 1 -- 172.21.15.27:0/3321459034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9160070430 msgr2=0x7f91600fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.720+0000 7f91685bb700 1 --2- 172.21.15.27:0/3321459034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9160070430 0x7f91600fc360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f915000b190 tx=0x7f9150006ee0).stop 2023-11-30T21:44:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.720+0000 7f91685bb700 1 -- 172.21.15.27:0/3321459034 shutdown_connections 2023-11-30T21:44:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.720+0000 7f91685bb700 1 --2- 172.21.15.27:0/3321459034 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f914802feb0 0x7f9148032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.720+0000 7f91685bb700 1 --2- 172.21.15.27:0/3321459034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9160070430 0x7f91600fc360 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.720+0000 7f91685bb700 1 -- 172.21.15.27:0/3321459034 >> 172.21.15.27:0/3321459034 conn(0x7f91600f63c0 msgr2=0x7f91600f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:07.722 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.720+0000 7f91685bb700 1 -- 172.21.15.27:0/3321459034 shutdown_connections 2023-11-30T21:44:07.723 INFO:teuthology.orchestra.run.smithi027.stderr:/usr/bin/ceph: 2023-11-30T21:44:07.720+0000 7f91685bb700 1 -- 172.21.15.27:0/3321459034 wait complete. 2023-11-30T21:44:08.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:07 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:06.442384+0000 mgr.smithi027.brrjcl (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-30T21:44:08.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:07 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:06.686785+0000 mon.smithi027 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:44:08.059 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-30T21:44:08.069 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: disabled 2023-11-30T21:44:08.069 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: systemctl is-active firewalld.service 2023-11-30T21:44:08.076 INFO:teuthology.orchestra.run.smithi027.stderr:systemctl: inactive 2023-11-30T21:44:08.077 INFO:teuthology.orchestra.run.smithi027.stderr:firewalld.service is not enabled 2023-11-30T21:44:08.078 INFO:teuthology.orchestra.run.smithi027.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-30T21:44:08.080 INFO:teuthology.orchestra.run.smithi027.stderr:Ceph Dashboard is now available at: 2023-11-30T21:44:08.080 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-11-30T21:44:08.080 INFO:teuthology.orchestra.run.smithi027.stderr: URL: https://smithi027.front.sepia.ceph.com:8443/ 2023-11-30T21:44:08.081 INFO:teuthology.orchestra.run.smithi027.stderr: User: admin 2023-11-30T21:44:08.084 INFO:teuthology.orchestra.run.smithi027.stderr: Password: d3k4mv71ul 2023-11-30T21:44:08.084 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-11-30T21:44:08.084 INFO:teuthology.orchestra.run.smithi027.stderr:You can access the Ceph CLI with: 2023-11-30T21:44:08.084 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-11-30T21:44:08.084 INFO:teuthology.orchestra.run.smithi027.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-30T21:44:08.085 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-11-30T21:44:08.085 INFO:teuthology.orchestra.run.smithi027.stderr:Please consider enabling telemetry to help improve Ceph: 2023-11-30T21:44:08.085 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-11-30T21:44:08.085 INFO:teuthology.orchestra.run.smithi027.stderr: ceph telemetry on 2023-11-30T21:44:08.085 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-11-30T21:44:08.085 INFO:teuthology.orchestra.run.smithi027.stderr:For more information see: 2023-11-30T21:44:08.085 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-11-30T21:44:08.086 INFO:teuthology.orchestra.run.smithi027.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-11-30T21:44:08.086 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-11-30T21:44:08.086 INFO:teuthology.orchestra.run.smithi027.stderr:Bootstrap complete. 2023-11-30T21:44:08.086 INFO:teuthology.orchestra.run.smithi027.stderr:Releasing lock 139937109555912 on /run/cephadm/6473e080-8fc9-11ee-95a2-87774f69a715.lock 2023-11-30T21:44:08.087 INFO:teuthology.orchestra.run.smithi027.stderr:Lock 139937109555912 released on /run/cephadm/6473e080-8fc9-11ee-95a2-87774f69a715.lock 2023-11-30T21:44:08.113 INFO:tasks.cephadm:Fetching config... 2023-11-30T21:44:08.114 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-11-30T21:44:08.114 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-30T21:44:08.173 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-30T21:44:08.173 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-11-30T21:44:08.173 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-30T21:44:08.233 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-30T21:44:08.233 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-11-30T21:44:08.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/mon.smithi027/keyring of=/dev/stdout 2023-11-30T21:44:08.300 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-30T21:44:08.301 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-11-30T21:44:08.301 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-30T21:44:08.359 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-30T21:44:08.360 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7p9dzOsiDJ7AAcUjutfye6E+r288LGQ7xmovbAfEysABv2cizir0UhGNWQ5M0xpAsJd+mGozG4KuyCuYNd/LcJ4pNZ0GA7B3MlDUDPAaw06K5vmthksG5JHhRQ0LnAsH/BDmggUAxDuKWxDk5dpzoFxZVoPalt0LUROqrcqUjPG01gxPCka7ZRCN6LzK8w4P0v2x65BHlsNWe9XeSyc23vEQ7k0tDL+URNMvzAzjr7SmA7nCUga73LxuuPAw1okdMjDgtDJp27Y0RQE9mJcxIeupCFlKKpCjvSaOc23rWWjxsZkz9t6Xpq/axB4OTuzY1aAEVjBRbWWOe5QAZwlADrIHFypvSMRLTxL6XTm81LyF9YTTkK+WKTinX3qb/hGySBPMv7D9zpnINlBdLtaKn1wlI3JusJIWPEwcevnBR7OFBOw+YWSpBpyWolyvrkAH9BePgFcXEQvEnkxSsrOWBJOfbd0QtYYZvsmm+f8Vg3+VfV4S4lK0sOsMO2XqHxP8= ceph-6473e080-8fc9-11ee-95a2-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-30T21:44:08.437 INFO:teuthology.orchestra.run.smithi027.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7p9dzOsiDJ7AAcUjutfye6E+r288LGQ7xmovbAfEysABv2cizir0UhGNWQ5M0xpAsJd+mGozG4KuyCuYNd/LcJ4pNZ0GA7B3MlDUDPAaw06K5vmthksG5JHhRQ0LnAsH/BDmggUAxDuKWxDk5dpzoFxZVoPalt0LUROqrcqUjPG01gxPCka7ZRCN6LzK8w4P0v2x65BHlsNWe9XeSyc23vEQ7k0tDL+URNMvzAzjr7SmA7nCUga73LxuuPAw1okdMjDgtDJp27Y0RQE9mJcxIeupCFlKKpCjvSaOc23rWWjxsZkz9t6Xpq/axB4OTuzY1aAEVjBRbWWOe5QAZwlADrIHFypvSMRLTxL6XTm81LyF9YTTkK+WKTinX3qb/hGySBPMv7D9zpnINlBdLtaKn1wlI3JusJIWPEwcevnBR7OFBOw+YWSpBpyWolyvrkAH9BePgFcXEQvEnkxSsrOWBJOfbd0QtYYZvsmm+f8Vg3+VfV4S4lK0sOsMO2XqHxP8= ceph-6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:44:08.448 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7p9dzOsiDJ7AAcUjutfye6E+r288LGQ7xmovbAfEysABv2cizir0UhGNWQ5M0xpAsJd+mGozG4KuyCuYNd/LcJ4pNZ0GA7B3MlDUDPAaw06K5vmthksG5JHhRQ0LnAsH/BDmggUAxDuKWxDk5dpzoFxZVoPalt0LUROqrcqUjPG01gxPCka7ZRCN6LzK8w4P0v2x65BHlsNWe9XeSyc23vEQ7k0tDL+URNMvzAzjr7SmA7nCUga73LxuuPAw1okdMjDgtDJp27Y0RQE9mJcxIeupCFlKKpCjvSaOc23rWWjxsZkz9t6Xpq/axB4OTuzY1aAEVjBRbWWOe5QAZwlADrIHFypvSMRLTxL6XTm81LyF9YTTkK+WKTinX3qb/hGySBPMv7D9zpnINlBdLtaKn1wlI3JusJIWPEwcevnBR7OFBOw+YWSpBpyWolyvrkAH9BePgFcXEQvEnkxSsrOWBJOfbd0QtYYZvsmm+f8Vg3+VfV4S4lK0sOsMO2XqHxP8= ceph-6473e080-8fc9-11ee-95a2-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-30T21:44:08.495 INFO:teuthology.orchestra.run.smithi049.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7p9dzOsiDJ7AAcUjutfye6E+r288LGQ7xmovbAfEysABv2cizir0UhGNWQ5M0xpAsJd+mGozG4KuyCuYNd/LcJ4pNZ0GA7B3MlDUDPAaw06K5vmthksG5JHhRQ0LnAsH/BDmggUAxDuKWxDk5dpzoFxZVoPalt0LUROqrcqUjPG01gxPCka7ZRCN6LzK8w4P0v2x65BHlsNWe9XeSyc23vEQ7k0tDL+URNMvzAzjr7SmA7nCUga73LxuuPAw1okdMjDgtDJp27Y0RQE9mJcxIeupCFlKKpCjvSaOc23rWWjxsZkz9t6Xpq/axB4OTuzY1aAEVjBRbWWOe5QAZwlADrIHFypvSMRLTxL6XTm81LyF9YTTkK+WKTinX3qb/hGySBPMv7D9zpnINlBdLtaKn1wlI3JusJIWPEwcevnBR7OFBOw+YWSpBpyWolyvrkAH9BePgFcXEQvEnkxSsrOWBJOfbd0QtYYZvsmm+f8Vg3+VfV4S4lK0sOsMO2XqHxP8= ceph-6473e080-8fc9-11ee-95a2-87774f69a715 2023-11-30T21:44:08.508 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-30T21:44:09.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:08 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:07.719385+0000 mon.smithi027 (mon.0) 106 : audit [DBG] from='client.? 172.21.15.27:0/3321459034' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-30T21:44:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.713+0000 7f65ea414700 1 -- 172.21.15.27:0/4168232051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65e4060180 msgr2=0x7f65e40f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.713+0000 7f65ea414700 1 --2- 172.21.15.27:0/4168232051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65e4060180 0x7f65e40f8f80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f65e0009fb0 tx=0x7f65e0009b30).stop 2023-11-30T21:44:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.713+0000 7f65ea414700 1 -- 172.21.15.27:0/4168232051 shutdown_connections 2023-11-30T21:44:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.713+0000 7f65ea414700 1 --2- 172.21.15.27:0/4168232051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65e4060180 0x7f65e40f8f80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.713+0000 7f65ea414700 1 -- 172.21.15.27:0/4168232051 >> 172.21.15.27:0/4168232051 conn(0x7f65e40f09a0 msgr2=0x7f65e40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.713+0000 7f65ea414700 1 -- 172.21.15.27:0/4168232051 shutdown_connections 2023-11-30T21:44:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.713+0000 7f65ea414700 1 -- 172.21.15.27:0/4168232051 wait complete. 2023-11-30T21:44:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.714+0000 7f65ea414700 1 Processor -- start 2023-11-30T21:44:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.714+0000 7f65ea414700 1 -- start start 2023-11-30T21:44:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.714+0000 7f65ea414700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65e4060180 0x7f65e40f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.714+0000 7f65ea414700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40f4e80 con 0x7f65e4060180 2023-11-30T21:44:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.714+0000 7f65e9412700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65e4060180 0x7f65e40f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.714+0000 7f65e9412700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65e4060180 0x7f65e40f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37370/0 (socket says 172.21.15.27:37370) 2023-11-30T21:44:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.714+0000 7f65e9412700 1 -- 172.21.15.27:0/1744304989 learned_addr learned my addr 172.21.15.27:0/1744304989 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:44:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.714+0000 7f65e9412700 1 -- 172.21.15.27:0/1744304989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e0005040 con 0x7f65e4060180 2023-11-30T21:44:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.715+0000 7f65e9412700 1 --2- 172.21.15.27:0/1744304989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65e4060180 0x7f65e40f6a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f65e0004dd0 tx=0x7f65e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.715+0000 7f65d5ffb700 1 -- 172.21.15.27:0/1744304989 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65e0009320 con 0x7f65e4060180 2023-11-30T21:44:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.715+0000 7f65d5ffb700 1 -- 172.21.15.27:0/1744304989 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f65e0013680 con 0x7f65e4060180 2023-11-30T21:44:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.715+0000 7f65ea414700 1 -- 172.21.15.27:0/1744304989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e40f5020 con 0x7f65e4060180 2023-11-30T21:44:09.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.715+0000 7f65d5ffb700 1 -- 172.21.15.27:0/1744304989 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65e001ecb0 con 0x7f65e4060180 2023-11-30T21:44:09.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.715+0000 7f65ea414700 1 -- 172.21.15.27:0/1744304989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e40f54b0 con 0x7f65e4060180 2023-11-30T21:44:09.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.716+0000 7f65d5ffb700 1 -- 172.21.15.27:0/1744304989 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f65e0014420 con 0x7f65e4060180 2023-11-30T21:44:09.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.716+0000 7f65d5ffb700 1 --2- 172.21.15.27:0/1744304989 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f65d002feb0 0x7f65d0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:09.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.716+0000 7f65d5ffb700 1 -- 172.21.15.27:0/1744304989 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f65e0040cb0 con 0x7f65e4060180 2023-11-30T21:44:09.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.716+0000 7f65ea414700 1 -- 172.21.15.27:0/1744304989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e4059100 con 0x7f65e4060180 2023-11-30T21:44:09.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.716+0000 7f65e8c11700 1 --2- 172.21.15.27:0/1744304989 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f65d002feb0 0x7f65d0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:09.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.716+0000 7f65e8c11700 1 --2- 172.21.15.27:0/1744304989 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f65d002feb0 0x7f65d0032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f65d80096a0 tx=0x7f65d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:09.721 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.720+0000 7f65d5ffb700 1 -- 172.21.15.27:0/1744304989 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f65e001c910 con 0x7f65e4060180 2023-11-30T21:44:09.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.880+0000 7f65ea414700 1 -- 172.21.15.27:0/1744304989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f65e40fbe50 con 0x7f65e4060180 2023-11-30T21:44:09.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.883+0000 7f65d5ffb700 1 -- 172.21.15.27:0/1744304989 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7f65e0004490 con 0x7f65e4060180 2023-11-30T21:44:09.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.885+0000 7f65ea414700 1 -- 172.21.15.27:0/1744304989 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f65d002feb0 msgr2=0x7f65d0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:09.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.885+0000 7f65ea414700 1 --2- 172.21.15.27:0/1744304989 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f65d002feb0 0x7f65d0032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f65d80096a0 tx=0x7f65d8006b40).stop 2023-11-30T21:44:09.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.885+0000 7f65ea414700 1 -- 172.21.15.27:0/1744304989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65e4060180 msgr2=0x7f65e40f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:09.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.885+0000 7f65ea414700 1 --2- 172.21.15.27:0/1744304989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65e4060180 0x7f65e40f6a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f65e0004dd0 tx=0x7f65e000b040).stop 2023-11-30T21:44:09.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.885+0000 7f65ea414700 1 -- 172.21.15.27:0/1744304989 shutdown_connections 2023-11-30T21:44:09.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.885+0000 7f65ea414700 1 --2- 172.21.15.27:0/1744304989 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f65d002feb0 0x7f65d0032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:09.888 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.885+0000 7f65ea414700 1 --2- 172.21.15.27:0/1744304989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65e4060180 0x7f65e40f6a00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:09.888 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.885+0000 7f65ea414700 1 -- 172.21.15.27:0/1744304989 >> 172.21.15.27:0/1744304989 conn(0x7f65e40f09a0 msgr2=0x7f65e40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:09.888 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.885+0000 7f65ea414700 1 -- 172.21.15.27:0/1744304989 shutdown_connections 2023-11-30T21:44:09.888 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:09.885+0000 7f65ea414700 1 -- 172.21.15.27:0/1744304989 wait complete. 2023-11-30T21:44:10.356 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi049 2023-11-30T21:44:10.356 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-30T21:44:10.356 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/etc/ceph/ceph.conf 2023-11-30T21:44:10.379 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2023-11-30T21:44:10.379 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-30T21:44:10.442 INFO:tasks.cephadm:Adding host smithi049 to orchestrator... 2023-11-30T21:44:10.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph orch host add smithi049 2023-11-30T21:44:11.165 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:10 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:09.884364+0000 mon.smithi027 (mon.0) 107 : audit [INF] from='client.? 172.21.15.27:0/1744304989' entity='client.admin' 2023-11-30T21:44:11.165 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:10 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:10.049320+0000 mon.smithi027 (mon.0) 108 : cluster [DBG] mgrmap e16: smithi027.brrjcl(active, since 6s) 2023-11-30T21:44:13.194 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:12 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:11.780650+0000 mon.smithi027 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:44:13.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:12 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:12.005699+0000 mon.smithi027 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:44:15.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:15 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:14.283185+0000 mon.smithi027 (mon.0) 111 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:44:18.261 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:16.880127+0000 mon.smithi027 (mon.0) 112 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:44:18.261 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:17.170266+0000 mon.smithi027 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:44:18.261 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:17.172271+0000 mon.smithi027 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:44:18.261 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:44:17.178447+0000 mgr.smithi027.brrjcl (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi027 on smithi027 2023-11-30T21:44:25.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:23.458243+0000 mgr.smithi027.brrjcl (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:27.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:27 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:25.458484+0000 mgr.smithi027.brrjcl (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:29.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:27.458835+0000 mgr.smithi027.brrjcl (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:31.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:29.459112+0000 mgr.smithi027.brrjcl (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:33.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:33 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:31.459410+0000 mgr.smithi027.brrjcl (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:35.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:35 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:33.459639+0000 mgr.smithi027.brrjcl (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:37.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:37 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:35.460017+0000 mgr.smithi027.brrjcl (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:38.374 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.372+0000 7f81e359e700 1 -- 172.21.15.49:0/2279842615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81e40f51a0 msgr2=0x7f81e40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:38.374 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.372+0000 7f81e359e700 1 --2- 172.21.15.49:0/2279842615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81e40f51a0 0x7f81e40f55c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f81d4009fb0 tx=0x7f81d4009b30).stop 2023-11-30T21:44:38.374 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.372+0000 7f81e359e700 1 -- 172.21.15.49:0/2279842615 shutdown_connections 2023-11-30T21:44:38.374 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.372+0000 7f81e359e700 1 --2- 172.21.15.49:0/2279842615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81e40f51a0 0x7f81e40f55c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:38.374 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.372+0000 7f81e359e700 1 -- 172.21.15.49:0/2279842615 >> 172.21.15.49:0/2279842615 conn(0x7f81e40f09a0 msgr2=0x7f81e40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:38.375 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.372+0000 7f81e359e700 1 -- 172.21.15.49:0/2279842615 shutdown_connections 2023-11-30T21:44:38.375 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.372+0000 7f81e359e700 1 -- 172.21.15.49:0/2279842615 wait complete. 2023-11-30T21:44:38.375 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.373+0000 7f81e359e700 1 Processor -- start 2023-11-30T21:44:38.375 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.373+0000 7f81e359e700 1 -- start start 2023-11-30T21:44:38.375 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.373+0000 7f81e359e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81e40f51a0 0x7f81e41091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:38.375 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.373+0000 7f81e359e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e4060220 con 0x7f81e40f51a0 2023-11-30T21:44:38.375 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.373+0000 7f81e259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81e40f51a0 0x7f81e41091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:38.375 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.374+0000 7f81e259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81e40f51a0 0x7f81e41091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:35096/0 (socket says 172.21.15.49:35096) 2023-11-30T21:44:38.375 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.374+0000 7f81e259c700 1 -- 172.21.15.49:0/2372945541 learned_addr learned my addr 172.21.15.49:0/2372945541 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:44:38.375 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.374+0000 7f81e259c700 1 -- 172.21.15.49:0/2372945541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d4005040 con 0x7f81e40f51a0 2023-11-30T21:44:38.377 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.374+0000 7f81e259c700 1 --2- 172.21.15.49:0/2372945541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81e40f51a0 0x7f81e41091d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f81d4013510 tx=0x7f81d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:38.377 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.374+0000 7f81daffd700 1 -- 172.21.15.49:0/2372945541 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81d4009320 con 0x7f81e40f51a0 2023-11-30T21:44:38.377 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.374+0000 7f81daffd700 1 -- 172.21.15.49:0/2372945541 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81d4013f00 con 0x7f81e40f51a0 2023-11-30T21:44:38.377 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.374+0000 7f81e359e700 1 -- 172.21.15.49:0/2372945541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e4060420 con 0x7f81e40f51a0 2023-11-30T21:44:38.377 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.374+0000 7f81daffd700 1 -- 172.21.15.49:0/2372945541 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81d4014dc0 con 0x7f81e40f51a0 2023-11-30T21:44:38.377 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.374+0000 7f81e359e700 1 -- 172.21.15.49:0/2372945541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e4109880 con 0x7f81e40f51a0 2023-11-30T21:44:38.377 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.375+0000 7f81daffd700 1 -- 172.21.15.49:0/2372945541 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81d401d3d0 con 0x7f81e40f51a0 2023-11-30T21:44:38.378 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.375+0000 7f81e359e700 1 -- 172.21.15.49:0/2372945541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e4059100 con 0x7f81e40f51a0 2023-11-30T21:44:38.378 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.375+0000 7f81daffd700 1 --2- 172.21.15.49:0/2372945541 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f81d002ff60 0x7f81d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:38.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.375+0000 7f81daffd700 1 -- 172.21.15.49:0/2372945541 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f81d40413b0 con 0x7f81e40f51a0 2023-11-30T21:44:38.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.376+0000 7f81e1d9b700 1 --2- 172.21.15.49:0/2372945541 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f81d002ff60 0x7f81d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:38.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.376+0000 7f81e1d9b700 1 --2- 172.21.15.49:0/2372945541 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f81d002ff60 0x7f81d0032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f81cc0096a0 tx=0x7f81cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:38.381 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.380+0000 7f81daffd700 1 -- 172.21.15.49:0/2372945541 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81d401cad0 con 0x7f81e40f51a0 2023-11-30T21:44:38.563 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:38.562+0000 7f81e359e700 1 -- 172.21.15.49:0/2372945541 --> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi049", "target": ["mon-mgr", ""]}) v1 -- 0x7f81e40f1b70 con 0x7f81d002ff60 2023-11-30T21:44:39.188 INFO:teuthology.orchestra.run.smithi049.stdout:Added host 'smithi049' 2023-11-30T21:44:39.189 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:39.187+0000 7f81daffd700 1 -- 172.21.15.49:0/2372945541 <== mgr.14162 v2:172.21.15.27:6800/4041101911 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f81e40f1b70 con 0x7f81d002ff60 2023-11-30T21:44:39.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:39.189+0000 7f81e359e700 1 -- 172.21.15.49:0/2372945541 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f81d002ff60 msgr2=0x7f81d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:39.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:39.189+0000 7f81e359e700 1 --2- 172.21.15.49:0/2372945541 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f81d002ff60 0x7f81d0032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f81cc0096a0 tx=0x7f81cc006b40).stop 2023-11-30T21:44:39.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:39.189+0000 7f81e359e700 1 -- 172.21.15.49:0/2372945541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81e40f51a0 msgr2=0x7f81e41091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:39.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:39.189+0000 7f81e359e700 1 --2- 172.21.15.49:0/2372945541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81e40f51a0 0x7f81e41091d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f81d4013510 tx=0x7f81d400b040).stop 2023-11-30T21:44:39.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:39.189+0000 7f81e359e700 1 -- 172.21.15.49:0/2372945541 shutdown_connections 2023-11-30T21:44:39.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:39.189+0000 7f81e359e700 1 --2- 172.21.15.49:0/2372945541 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f81d002ff60 0x7f81d0032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:39.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:39.189+0000 7f81e359e700 1 --2- 172.21.15.49:0/2372945541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81e40f51a0 0x7f81e41091d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:39.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:39.189+0000 7f81e359e700 1 -- 172.21.15.49:0/2372945541 >> 172.21.15.49:0/2372945541 conn(0x7f81e40f09a0 msgr2=0x7f81e40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:39.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:39.189+0000 7f81e359e700 1 -- 172.21.15.49:0/2372945541 shutdown_connections 2023-11-30T21:44:39.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:39.189+0000 7f81e359e700 1 -- 172.21.15.49:0/2372945541 wait complete. 2023-11-30T21:44:39.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:37.460313+0000 mgr.smithi027.brrjcl (mgr.14162) 13 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:39.519 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph orch host ls --format=json 2023-11-30T21:44:40.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:40 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:38.563934+0000 mgr.smithi027.brrjcl (mgr.14162) 14 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi049", "target": ["mon-mgr", ""]}]: dispatch 2023-11-30T21:44:40.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:40 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:39.186494+0000 mon.smithi027 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:44:40.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:40 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:44:39.186870+0000 mgr.smithi027.brrjcl (mgr.14162) 15 : cephadm [INF] Added host smithi049 2023-11-30T21:44:40.614 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.612+0000 7fc7b4abf700 1 -- 172.21.15.49:0/317954707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b00f51a0 msgr2=0x7fc7b00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:45.287 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.612+0000 7fc7b4abf700 1 --2- 172.21.15.49:0/317954707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b00f51a0 0x7fc7b00f55c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fc7a0009fb0 tx=0x7fc7a0009b30).stop 2023-11-30T21:44:45.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.612+0000 7fc7b4abf700 1 -- 172.21.15.49:0/317954707 shutdown_connections 2023-11-30T21:44:45.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.612+0000 7fc7b4abf700 1 --2- 172.21.15.49:0/317954707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b00f51a0 0x7fc7b00f55c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:45.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.612+0000 7fc7b4abf700 1 -- 172.21.15.49:0/317954707 >> 172.21.15.49:0/317954707 conn(0x7fc7b00f09a0 msgr2=0x7fc7b00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:45.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.613+0000 7fc7b4abf700 1 -- 172.21.15.49:0/317954707 shutdown_connections 2023-11-30T21:44:45.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.613+0000 7fc7b4abf700 1 -- 172.21.15.49:0/317954707 wait complete. 2023-11-30T21:44:45.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.613+0000 7fc7b4abf700 1 Processor -- start 2023-11-30T21:44:45.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.613+0000 7fc7b4abf700 1 -- start start 2023-11-30T21:44:45.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.613+0000 7fc7b4abf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b00f51a0 0x7fc7b01064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:45.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.613+0000 7fc7b4abf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b01069d0 con 0x7fc7b00f51a0 2023-11-30T21:44:45.289 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:41 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:39.460655+0000 mgr.smithi027.brrjcl (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:45.289 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:42 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:40.803287+0000 mgr.smithi027.brrjcl (mgr.14162) 17 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-30T21:44:45.289 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:41.460959+0000 mgr.smithi027.brrjcl (mgr.14162) 18 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:45.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.614+0000 7fc7af7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b00f51a0 0x7fc7b01064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:45.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.614+0000 7fc7af7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b00f51a0 0x7fc7b01064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:36096/0 (socket says 172.21.15.49:36096) 2023-11-30T21:44:45.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.614+0000 7fc7af7fe700 1 -- 172.21.15.49:0/676353729 learned_addr learned my addr 172.21.15.49:0/676353729 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:44:45.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.614+0000 7fc7af7fe700 1 -- 172.21.15.49:0/676353729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a0005040 con 0x7fc7b00f51a0 2023-11-30T21:44:45.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.614+0000 7fc7af7fe700 1 --2- 172.21.15.49:0/676353729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b00f51a0 0x7fc7b01064e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fc7a0013510 tx=0x7fc7a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:45.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.615+0000 7fc797fff700 1 -- 172.21.15.49:0/676353729 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7a0009320 con 0x7fc7b00f51a0 2023-11-30T21:44:45.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.615+0000 7fc7b4abf700 1 -- 172.21.15.49:0/676353729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b0106bd0 con 0x7fc7b00f51a0 2023-11-30T21:44:45.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.615+0000 7fc797fff700 1 -- 172.21.15.49:0/676353729 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7a0013f00 con 0x7fc7b00f51a0 2023-11-30T21:44:45.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.615+0000 7fc797fff700 1 -- 172.21.15.49:0/676353729 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7a0014dc0 con 0x7fc7b00f51a0 2023-11-30T21:44:45.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.615+0000 7fc7b4abf700 1 -- 172.21.15.49:0/676353729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b0107060 con 0x7fc7b00f51a0 2023-11-30T21:44:45.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.615+0000 7fc797fff700 1 -- 172.21.15.49:0/676353729 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7a0014420 con 0x7fc7b00f51a0 2023-11-30T21:44:45.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.615+0000 7fc7b4abf700 1 -- 172.21.15.49:0/676353729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b0045ad0 con 0x7fc7b00f51a0 2023-11-30T21:44:45.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.616+0000 7fc797fff700 1 --2- 172.21.15.49:0/676353729 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc79802ff60 0x7fc798032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:45.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.616+0000 7fc797fff700 1 -- 172.21.15.49:0/676353729 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7a0041110 con 0x7fc7b00f51a0 2023-11-30T21:44:45.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.616+0000 7fc7aeffd700 1 --2- 172.21.15.49:0/676353729 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc79802ff60 0x7fc798032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:45.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.616+0000 7fc7aeffd700 1 --2- 172.21.15.49:0/676353729 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc79802ff60 0x7fc798032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc7a40096a0 tx=0x7fc7a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:45.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.620+0000 7fc797fff700 1 -- 172.21.15.49:0/676353729 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7a00095f0 con 0x7fc7b00f51a0 2023-11-30T21:44:45.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.802+0000 7fc7b4abf700 1 -- 172.21.15.49:0/676353729 --> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc7b00f2790 con 0x7fc79802ff60 2023-11-30T21:44:45.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.803+0000 7fc797fff700 1 -- 172.21.15.49:0/676353729 <== mgr.14162 v2:172.21.15.27:6800/4041101911 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7fc7b00f2790 con 0x7fc79802ff60 2023-11-30T21:44:45.293 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:44:45.293 INFO:teuthology.orchestra.run.smithi049.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "smithi049", "hostname": "smithi049", "labels": [], "status": ""}] 2023-11-30T21:44:45.294 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.805+0000 7fc7b4abf700 1 -- 172.21.15.49:0/676353729 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc79802ff60 msgr2=0x7fc798032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:45.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.805+0000 7fc7b4abf700 1 --2- 172.21.15.49:0/676353729 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc79802ff60 0x7fc798032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc7a40096a0 tx=0x7fc7a4006b40).stop 2023-11-30T21:44:45.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.805+0000 7fc7b4abf700 1 -- 172.21.15.49:0/676353729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b00f51a0 msgr2=0x7fc7b01064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:45.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.805+0000 7fc7b4abf700 1 --2- 172.21.15.49:0/676353729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b00f51a0 0x7fc7b01064e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fc7a0013510 tx=0x7fc7a000b040).stop 2023-11-30T21:44:45.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.805+0000 7fc7b4abf700 1 -- 172.21.15.49:0/676353729 shutdown_connections 2023-11-30T21:44:45.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.805+0000 7fc7b4abf700 1 --2- 172.21.15.49:0/676353729 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc79802ff60 0x7fc798032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:45.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.805+0000 7fc7b4abf700 1 --2- 172.21.15.49:0/676353729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b00f51a0 0x7fc7b01064e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:45.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.805+0000 7fc7b4abf700 1 -- 172.21.15.49:0/676353729 >> 172.21.15.49:0/676353729 conn(0x7fc7b00f09a0 msgr2=0x7fc7b005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:45.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.805+0000 7fc7b4abf700 1 -- 172.21.15.49:0/676353729 shutdown_connections 2023-11-30T21:44:45.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:40.805+0000 7fc7b4abf700 1 -- 172.21.15.49:0/676353729 wait complete. 2023-11-30T21:44:45.296 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-30T21:44:45.297 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph osd crush tunables default 2023-11-30T21:44:45.331 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:45 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:43.461324+0000 mgr.smithi027.brrjcl (mgr.14162) 19 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.503+0000 7f276d890700 1 -- 172.21.15.27:0/1768535072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27680f51a0 msgr2=0x7f27680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:46.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.503+0000 7f276d890700 1 --2- 172.21.15.27:0/1768535072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27680f51a0 0x7f27680f55c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f275c009fb0 tx=0x7f275c009b30).stop 2023-11-30T21:44:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.503+0000 7f276d890700 1 -- 172.21.15.27:0/1768535072 shutdown_connections 2023-11-30T21:44:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.503+0000 7f276d890700 1 --2- 172.21.15.27:0/1768535072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27680f51a0 0x7f27680f55c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.503+0000 7f276d890700 1 -- 172.21.15.27:0/1768535072 >> 172.21.15.27:0/1768535072 conn(0x7f27680f09a0 msgr2=0x7f27680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.503+0000 7f276d890700 1 -- 172.21.15.27:0/1768535072 shutdown_connections 2023-11-30T21:44:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.503+0000 7f276d890700 1 -- 172.21.15.27:0/1768535072 wait complete. 2023-11-30T21:44:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.503+0000 7f276d890700 1 Processor -- start 2023-11-30T21:44:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.503+0000 7f276d890700 1 -- start start 2023-11-30T21:44:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.504+0000 7f276d890700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27680f51a0 0x7f27681064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.504+0000 7f276d890700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27681069d0 con 0x7f27680f51a0 2023-11-30T21:44:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.504+0000 7f276c88e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27680f51a0 0x7f27681064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.504+0000 7f276c88e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27680f51a0 0x7f27681064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34850/0 (socket says 172.21.15.27:34850) 2023-11-30T21:44:46.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.504+0000 7f276c88e700 1 -- 172.21.15.27:0/2820294099 learned_addr learned my addr 172.21.15.27:0/2820294099 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-11-30T21:44:46.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.504+0000 7f276c88e700 1 -- 172.21.15.27:0/2820294099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275c005040 con 0x7f27680f51a0 2023-11-30T21:44:46.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.504+0000 7f276c88e700 1 --2- 172.21.15.27:0/2820294099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27680f51a0 0x7f27681064e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f275c013510 tx=0x7f275c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:46.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.504+0000 7f27657fa700 1 -- 172.21.15.27:0/2820294099 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f275c009320 con 0x7f27680f51a0 2023-11-30T21:44:46.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.505+0000 7f276d890700 1 -- 172.21.15.27:0/2820294099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2768106bd0 con 0x7f27680f51a0 2023-11-30T21:44:46.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.505+0000 7f27657fa700 1 -- 172.21.15.27:0/2820294099 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f275c013f00 con 0x7f27680f51a0 2023-11-30T21:44:46.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.505+0000 7f27657fa700 1 -- 172.21.15.27:0/2820294099 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f275c014dc0 con 0x7f27680f51a0 2023-11-30T21:44:46.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.505+0000 7f276d890700 1 -- 172.21.15.27:0/2820294099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2768107060 con 0x7f27680f51a0 2023-11-30T21:44:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.505+0000 7f276d890700 1 -- 172.21.15.27:0/2820294099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2768045ad0 con 0x7f27680f51a0 2023-11-30T21:44:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.506+0000 7f27657fa700 1 -- 172.21.15.27:0/2820294099 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f275c0095f0 con 0x7f27680f51a0 2023-11-30T21:44:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.506+0000 7f27657fa700 1 --2- 172.21.15.27:0/2820294099 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f275802ff60 0x7f2758032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.506+0000 7f27657fa700 1 -- 172.21.15.27:0/2820294099 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f275c041000 con 0x7f27680f51a0 2023-11-30T21:44:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.506+0000 7f2767fff700 1 --2- 172.21.15.27:0/2820294099 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f275802ff60 0x7f2758032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:46.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.506+0000 7f2767fff700 1 --2- 172.21.15.27:0/2820294099 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f275802ff60 0x7f2758032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f27540096a0 tx=0x7f2754006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.510+0000 7f27657fa700 1 -- 172.21.15.27:0/2820294099 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f275c014420 con 0x7f27680f51a0 2023-11-30T21:44:46.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:46.675+0000 7f276d890700 1 -- 172.21.15.27:0/2820294099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f2768107f10 con 0x7f27680f51a0 2023-11-30T21:44:47.056 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:47.055+0000 7f27657fa700 1 -- 172.21.15.27:0/2820294099 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f275c010230 con 0x7f27680f51a0 2023-11-30T21:44:47.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:47.057+0000 7f276d890700 1 -- 172.21.15.27:0/2820294099 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f275802ff60 msgr2=0x7f2758032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:47.059 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:47.057+0000 7f276d890700 1 --2- 172.21.15.27:0/2820294099 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f275802ff60 0x7f2758032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f27540096a0 tx=0x7f2754006b40).stop 2023-11-30T21:44:47.059 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:47.057+0000 7f276d890700 1 -- 172.21.15.27:0/2820294099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27680f51a0 msgr2=0x7f27681064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:47.059 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:47.057+0000 7f276d890700 1 --2- 172.21.15.27:0/2820294099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27680f51a0 0x7f27681064e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f275c013510 tx=0x7f275c00b040).stop 2023-11-30T21:44:47.059 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:47.057+0000 7f276d890700 1 -- 172.21.15.27:0/2820294099 shutdown_connections 2023-11-30T21:44:47.060 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:47.057+0000 7f276d890700 1 --2- 172.21.15.27:0/2820294099 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f275802ff60 0x7f2758032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:47.060 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:47.057+0000 7f276d890700 1 --2- 172.21.15.27:0/2820294099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27680f51a0 0x7f27681064e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:47.060 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:47.057+0000 7f276d890700 1 -- 172.21.15.27:0/2820294099 >> 172.21.15.27:0/2820294099 conn(0x7f27680f09a0 msgr2=0x7f276805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:47.060 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:47.057+0000 7f276d890700 1 -- 172.21.15.27:0/2820294099 shutdown_connections 2023-11-30T21:44:47.061 INFO:teuthology.orchestra.run.smithi027.stderr:2023-11-30T21:44:47.057+0000 7f276d890700 1 -- 172.21.15.27:0/2820294099 wait complete. 2023-11-30T21:44:47.061 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2023-11-30T21:44:47.382 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:47 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:45.461601+0000 mgr.smithi027.brrjcl (mgr.14162) 20 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:47.383 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:47 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:46.676819+0000 mon.smithi027 (mon.0) 116 : audit [INF] from='client.? 172.21.15.27:0/2820294099' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-30T21:44:47.425 INFO:tasks.cephadm:Adding mon.smithi027 on smithi027 2023-11-30T21:44:47.425 INFO:tasks.cephadm:Adding mon.smithi049 on smithi049 2023-11-30T21:44:47.426 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph orch apply mon '2;smithi027:172.21.15.27=smithi027;smithi049:172.21.15.49=smithi049' 2023-11-30T21:44:48.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:48 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:47.056307+0000 mon.smithi027 (mon.0) 117 : audit [INF] from='client.? 172.21.15.27:0/2820294099' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-30T21:44:48.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:48 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:47.056352+0000 mon.smithi027 (mon.0) 118 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-30T21:44:48.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.555+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3132328477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d380f51c0 msgr2=0x7f3d380f55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:48.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.555+0000 7f3d3d64e700 1 --2- 172.21.15.49:0/3132328477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d380f51c0 0x7f3d380f55e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f3d28009fb0 tx=0x7f3d28009b30).stop 2023-11-30T21:44:48.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.555+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3132328477 shutdown_connections 2023-11-30T21:44:48.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.555+0000 7f3d3d64e700 1 --2- 172.21.15.49:0/3132328477 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d380f51c0 0x7f3d380f55e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:48.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.555+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3132328477 >> 172.21.15.49:0/3132328477 conn(0x7f3d380f0920 msgr2=0x7f3d380f2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:48.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.555+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3132328477 shutdown_connections 2023-11-30T21:44:48.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.555+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3132328477 wait complete. 2023-11-30T21:44:48.558 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.555+0000 7f3d3d64e700 1 Processor -- start 2023-11-30T21:44:48.558 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.555+0000 7f3d3d64e700 1 -- start start 2023-11-30T21:44:48.558 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.556+0000 7f3d3d64e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d380f51c0 0x7f3d38109200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:48.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.556+0000 7f3d3d64e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d38060280 con 0x7f3d380f51c0 2023-11-30T21:44:48.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.556+0000 7f3d36ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d380f51c0 0x7f3d38109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:48.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.556+0000 7f3d36ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d380f51c0 0x7f3d38109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:58360/0 (socket says 172.21.15.49:58360) 2023-11-30T21:44:48.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.556+0000 7f3d36ffd700 1 -- 172.21.15.49:0/3758489143 learned_addr learned my addr 172.21.15.49:0/3758489143 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:44:48.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.556+0000 7f3d36ffd700 1 -- 172.21.15.49:0/3758489143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d28005040 con 0x7f3d380f51c0 2023-11-30T21:44:48.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.556+0000 7f3d36ffd700 1 --2- 172.21.15.49:0/3758489143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d380f51c0 0x7f3d38109200 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f3d28013510 tx=0x7f3d2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:48.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.556+0000 7f3d1f7fe700 1 -- 172.21.15.49:0/3758489143 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d28009320 con 0x7f3d380f51c0 2023-11-30T21:44:48.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.557+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3758489143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d38060480 con 0x7f3d380f51c0 2023-11-30T21:44:48.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.557+0000 7f3d1f7fe700 1 -- 172.21.15.49:0/3758489143 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d28013f00 con 0x7f3d380f51c0 2023-11-30T21:44:48.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.557+0000 7f3d1f7fe700 1 -- 172.21.15.49:0/3758489143 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d28014d90 con 0x7f3d380f51c0 2023-11-30T21:44:48.559 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.557+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3758489143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d38109930 con 0x7f3d380f51c0 2023-11-30T21:44:48.560 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.557+0000 7f3d1f7fe700 1 -- 172.21.15.49:0/3758489143 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d280263a0 con 0x7f3d380f51c0 2023-11-30T21:44:48.560 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.557+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3758489143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d38045ad0 con 0x7f3d380f51c0 2023-11-30T21:44:48.560 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.558+0000 7f3d1f7fe700 1 --2- 172.21.15.49:0/3758489143 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f3d2002ff60 0x7f3d20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:48.560 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.558+0000 7f3d1f7fe700 1 -- 172.21.15.49:0/3758489143 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d28041300 con 0x7f3d380f51c0 2023-11-30T21:44:48.560 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.558+0000 7f3d367fc700 1 --2- 172.21.15.49:0/3758489143 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f3d2002ff60 0x7f3d20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:48.561 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.558+0000 7f3d367fc700 1 --2- 172.21.15.49:0/3758489143 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f3d2002ff60 0x7f3d20032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f3d2c0096a0 tx=0x7f3d2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:48.563 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.562+0000 7f3d1f7fe700 1 -- 172.21.15.49:0/3758489143 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d280126f0 con 0x7f3d380f51c0 2023-11-30T21:44:48.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.750+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3758489143 --> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi027:172.21.15.27=smithi027;smithi049:172.21.15.49=smithi049", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d380f1c80 con 0x7f3d2002ff60 2023-11-30T21:44:48.757 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.756+0000 7f3d1f7fe700 1 -- 172.21.15.49:0/3758489143 <== mgr.14162 v2:172.21.15.27:6800/4041101911 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3d380f1c80 con 0x7f3d2002ff60 2023-11-30T21:44:48.757 INFO:teuthology.orchestra.run.smithi049.stdout:Scheduled mon update... 2023-11-30T21:44:48.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.757+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3758489143 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f3d2002ff60 msgr2=0x7f3d20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:48.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.757+0000 7f3d3d64e700 1 --2- 172.21.15.49:0/3758489143 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f3d2002ff60 0x7f3d20032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f3d2c0096a0 tx=0x7f3d2c006b40).stop 2023-11-30T21:44:48.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.757+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3758489143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d380f51c0 msgr2=0x7f3d38109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:48.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.757+0000 7f3d3d64e700 1 --2- 172.21.15.49:0/3758489143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d380f51c0 0x7f3d38109200 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f3d28013510 tx=0x7f3d2800b040).stop 2023-11-30T21:44:48.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.757+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3758489143 shutdown_connections 2023-11-30T21:44:48.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.757+0000 7f3d3d64e700 1 --2- 172.21.15.49:0/3758489143 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f3d2002ff60 0x7f3d20032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:48.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.757+0000 7f3d3d64e700 1 --2- 172.21.15.49:0/3758489143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d380f51c0 0x7f3d38109200 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:48.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.757+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3758489143 >> 172.21.15.49:0/3758489143 conn(0x7f3d380f0920 msgr2=0x7f3d380f14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:48.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.757+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3758489143 shutdown_connections 2023-11-30T21:44:48.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:48.758+0000 7f3d3d64e700 1 -- 172.21.15.49:0/3758489143 wait complete. 2023-11-30T21:44:49.232 DEBUG:teuthology.orchestra.run.smithi049:mon.smithi049> sudo journalctl -f -n 0 -u ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi049.service 2023-11-30T21:44:49.235 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:44:49.236 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:44:49.265 INFO:journalctl@ceph.mon.smithi049.smithi049.stdout:-- Logs begin at Thu 2023-11-30 21:30:39 UTC. -- 2023-11-30T21:44:49.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:49 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:47.461994+0000 mgr.smithi027.brrjcl (mgr.14162) 21 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:49.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:49 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:48.756852+0000 mon.smithi027 (mon.0) 119 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:44:50.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.470+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/3079129341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9f00f4820 msgr2=0x7fa9f00f4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:50.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.470+0000 7fa9f4bb5700 1 --2- 172.21.15.49:0/3079129341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9f00f4820 0x7fa9f00f4c40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fa9e0009fb0 tx=0x7fa9e0009b30).stop 2023-11-30T21:44:50.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.470+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/3079129341 shutdown_connections 2023-11-30T21:44:50.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.470+0000 7fa9f4bb5700 1 --2- 172.21.15.49:0/3079129341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9f00f4820 0x7fa9f00f4c40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:50.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.470+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/3079129341 >> 172.21.15.49:0/3079129341 conn(0x7fa9f00f0260 msgr2=0x7fa9f00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:50.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.470+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/3079129341 shutdown_connections 2023-11-30T21:44:50.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.470+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/3079129341 wait complete. 2023-11-30T21:44:50.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.471+0000 7fa9f4bb5700 1 Processor -- start 2023-11-30T21:44:50.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.471+0000 7fa9f4bb5700 1 -- start start 2023-11-30T21:44:50.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.471+0000 7fa9f4bb5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9f00f4820 0x7fa9f01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:50.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.471+0000 7fa9f4bb5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f0106bd0 con 0x7fa9f00f4820 2023-11-30T21:44:50.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.471+0000 7fa9ee59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9f00f4820 0x7fa9f01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:50.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.471+0000 7fa9ee59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9f00f4820 0x7fa9f01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:58380/0 (socket says 172.21.15.49:58380) 2023-11-30T21:44:50.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.471+0000 7fa9ee59c700 1 -- 172.21.15.49:0/2594183504 learned_addr learned my addr 172.21.15.49:0/2594183504 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:44:50.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.472+0000 7fa9ee59c700 1 -- 172.21.15.49:0/2594183504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e0005040 con 0x7fa9f00f4820 2023-11-30T21:44:50.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.472+0000 7fa9ee59c700 1 --2- 172.21.15.49:0/2594183504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9f00f4820 0x7fa9f01066e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fa9e0013510 tx=0x7fa9e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:50.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.472+0000 7fa9e6ffd700 1 -- 172.21.15.49:0/2594183504 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9e0009320 con 0x7fa9f00f4820 2023-11-30T21:44:50.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.472+0000 7fa9e6ffd700 1 -- 172.21.15.49:0/2594183504 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9e0013f00 con 0x7fa9f00f4820 2023-11-30T21:44:50.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.472+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/2594183504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f0106dd0 con 0x7fa9f00f4820 2023-11-30T21:44:50.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.472+0000 7fa9e6ffd700 1 -- 172.21.15.49:0/2594183504 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9e0014d90 con 0x7fa9f00f4820 2023-11-30T21:44:50.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.472+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/2594183504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f0107260 con 0x7fa9f00f4820 2023-11-30T21:44:50.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.473+0000 7fa9e6ffd700 1 -- 172.21.15.49:0/2594183504 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9e00095f0 con 0x7fa9f00f4820 2023-11-30T21:44:50.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.473+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/2594183504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f0045ad0 con 0x7fa9f00f4820 2023-11-30T21:44:50.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.473+0000 7fa9e6ffd700 1 --2- 172.21.15.49:0/2594183504 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa9dc02ffb0 0x7fa9dc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:50.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.473+0000 7fa9e6ffd700 1 -- 172.21.15.49:0/2594183504 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9e0040fb0 con 0x7fa9f00f4820 2023-11-30T21:44:50.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.473+0000 7fa9edd9b700 1 --2- 172.21.15.49:0/2594183504 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa9dc02ffb0 0x7fa9dc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:50.476 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.474+0000 7fa9edd9b700 1 --2- 172.21.15.49:0/2594183504 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa9dc02ffb0 0x7fa9dc032470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa9d80096a0 tx=0x7fa9d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:50.479 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.478+0000 7fa9e6ffd700 1 -- 172.21.15.49:0/2594183504 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9e006d2b0 con 0x7fa9f00f4820 2023-11-30T21:44:50.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:50 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:48.752674+0000 mgr.smithi027.brrjcl (mgr.14162) 22 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi027:172.21.15.27=smithi027;smithi049:172.21.15.49=smithi049", "target": ["mon-mgr", ""]}]: dispatch 2023-11-30T21:44:50.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:50 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:44:48.754093+0000 mgr.smithi027.brrjcl (mgr.14162) 23 : cephadm [INF] Saving service mon spec with placement smithi027:172.21.15.27=smithi027;smithi049:172.21.15.49=smithi049;count:2 2023-11-30T21:44:50.700 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.699+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/2594183504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9f0108110 con 0x7fa9f00f4820 2023-11-30T21:44:50.700 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.699+0000 7fa9e6ffd700 1 -- 172.21.15.49:0/2594183504 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa9e0014420 con 0x7fa9f00f4820 2023-11-30T21:44:50.701 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:44:50.701 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:44:50.702 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.701+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/2594183504 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa9dc02ffb0 msgr2=0x7fa9dc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:50.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.701+0000 7fa9f4bb5700 1 --2- 172.21.15.49:0/2594183504 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa9dc02ffb0 0x7fa9dc032470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fa9d80096a0 tx=0x7fa9d8006b40).stop 2023-11-30T21:44:50.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.701+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/2594183504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9f00f4820 msgr2=0x7fa9f01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:50.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.701+0000 7fa9f4bb5700 1 --2- 172.21.15.49:0/2594183504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9f00f4820 0x7fa9f01066e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fa9e0013510 tx=0x7fa9e000b040).stop 2023-11-30T21:44:50.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.701+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/2594183504 shutdown_connections 2023-11-30T21:44:50.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.701+0000 7fa9f4bb5700 1 --2- 172.21.15.49:0/2594183504 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa9dc02ffb0 0x7fa9dc032470 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:50.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.701+0000 7fa9f4bb5700 1 --2- 172.21.15.49:0/2594183504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9f00f4820 0x7fa9f01066e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:50.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.701+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/2594183504 >> 172.21.15.49:0/2594183504 conn(0x7fa9f00f0260 msgr2=0x7fa9f00f0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:50.704 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.701+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/2594183504 shutdown_connections 2023-11-30T21:44:50.704 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:50.701+0000 7fa9f4bb5700 1 -- 172.21.15.49:0/2594183504 wait complete. 2023-11-30T21:44:50.704 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:44:51.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:49.462377+0000 mgr.smithi027.brrjcl (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:51.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:50.700674+0000 mon.smithi027 (mon.0) 120 : audit [DBG] from='client.? 172.21.15.49:0/2594183504' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:44:52.072 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:44:52.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:44:53.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.223+0000 7f7ae3559700 1 -- 172.21.15.49:0/2957663420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7adc0f4740 msgr2=0x7f7adc0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:53.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.223+0000 7f7ae3559700 1 --2- 172.21.15.49:0/2957663420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7adc0f4740 0x7f7adc0f4b60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7ad8009fb0 tx=0x7f7ad8009b30).stop 2023-11-30T21:44:53.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.223+0000 7f7ae3559700 1 -- 172.21.15.49:0/2957663420 shutdown_connections 2023-11-30T21:44:53.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.223+0000 7f7ae3559700 1 --2- 172.21.15.49:0/2957663420 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7adc0f4740 0x7f7adc0f4b60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:53.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.223+0000 7f7ae3559700 1 -- 172.21.15.49:0/2957663420 >> 172.21.15.49:0/2957663420 conn(0x7f7adc0f0260 msgr2=0x7f7adc0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:53.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.223+0000 7f7ae3559700 1 -- 172.21.15.49:0/2957663420 shutdown_connections 2023-11-30T21:44:53.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.223+0000 7f7ae3559700 1 -- 172.21.15.49:0/2957663420 wait complete. 2023-11-30T21:44:53.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.223+0000 7f7ae3559700 1 Processor -- start 2023-11-30T21:44:53.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.223+0000 7f7ae3559700 1 -- start start 2023-11-30T21:44:53.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.223+0000 7f7ae3559700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7adc0f4740 0x7f7adc1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:53.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.223+0000 7f7ae3559700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc106bc0 con 0x7f7adc0f4740 2023-11-30T21:44:53.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.224+0000 7f7ae12f5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7adc0f4740 0x7f7adc1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:53.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.224+0000 7f7ae12f5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7adc0f4740 0x7f7adc1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:58392/0 (socket says 172.21.15.49:58392) 2023-11-30T21:44:53.227 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.224+0000 7f7ae12f5700 1 -- 172.21.15.49:0/839972332 learned_addr learned my addr 172.21.15.49:0/839972332 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:44:53.227 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.224+0000 7f7ae12f5700 1 -- 172.21.15.49:0/839972332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad8005040 con 0x7f7adc0f4740 2023-11-30T21:44:53.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.224+0000 7f7ae12f5700 1 --2- 172.21.15.49:0/839972332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7adc0f4740 0x7f7adc1066d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f7ad8013510 tx=0x7f7ad800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:53.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.224+0000 7f7acdffb700 1 -- 172.21.15.49:0/839972332 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ad8009320 con 0x7f7adc0f4740 2023-11-30T21:44:53.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.225+0000 7f7ae3559700 1 -- 172.21.15.49:0/839972332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7adc106dc0 con 0x7f7adc0f4740 2023-11-30T21:44:53.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.225+0000 7f7acdffb700 1 -- 172.21.15.49:0/839972332 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ad8013f00 con 0x7f7adc0f4740 2023-11-30T21:44:53.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.225+0000 7f7acdffb700 1 -- 172.21.15.49:0/839972332 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ad8014d90 con 0x7f7adc0f4740 2023-11-30T21:44:53.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.225+0000 7f7ae3559700 1 -- 172.21.15.49:0/839972332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7adc107250 con 0x7f7adc0f4740 2023-11-30T21:44:53.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.225+0000 7f7acdffb700 1 -- 172.21.15.49:0/839972332 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ad8014420 con 0x7f7adc0f4740 2023-11-30T21:44:53.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.226+0000 7f7ae3559700 1 -- 172.21.15.49:0/839972332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7adc045ad0 con 0x7f7adc0f4740 2023-11-30T21:44:53.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.226+0000 7f7acdffb700 1 --2- 172.21.15.49:0/839972332 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7ac802ff60 0x7f7ac8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:53.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.226+0000 7f7acdffb700 1 -- 172.21.15.49:0/839972332 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ad80410c0 con 0x7f7adc0f4740 2023-11-30T21:44:53.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.226+0000 7f7ae0af4700 1 --2- 172.21.15.49:0/839972332 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7ac802ff60 0x7f7ac8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:53.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.226+0000 7f7ae0af4700 1 --2- 172.21.15.49:0/839972332 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7ac802ff60 0x7f7ac8032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7ad00096a0 tx=0x7f7ad0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:53.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.230+0000 7f7acdffb700 1 -- 172.21.15.49:0/839972332 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ad80095f0 con 0x7f7adc0f4740 2023-11-30T21:44:53.451 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.450+0000 7f7ae3559700 1 -- 172.21.15.49:0/839972332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7adc108100 con 0x7f7adc0f4740 2023-11-30T21:44:53.451 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.450+0000 7f7acdffb700 1 -- 172.21.15.49:0/839972332 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ad801db70 con 0x7f7adc0f4740 2023-11-30T21:44:53.452 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:44:53.452 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:44:53.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.452+0000 7f7ae3559700 1 -- 172.21.15.49:0/839972332 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7ac802ff60 msgr2=0x7f7ac8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:53.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.452+0000 7f7ae3559700 1 --2- 172.21.15.49:0/839972332 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7ac802ff60 0x7f7ac8032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7ad00096a0 tx=0x7f7ad0006b40).stop 2023-11-30T21:44:53.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.452+0000 7f7ae3559700 1 -- 172.21.15.49:0/839972332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7adc0f4740 msgr2=0x7f7adc1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:53.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.452+0000 7f7ae3559700 1 --2- 172.21.15.49:0/839972332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7adc0f4740 0x7f7adc1066d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f7ad8013510 tx=0x7f7ad800b040).stop 2023-11-30T21:44:53.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.452+0000 7f7ae3559700 1 -- 172.21.15.49:0/839972332 shutdown_connections 2023-11-30T21:44:53.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.452+0000 7f7ae3559700 1 --2- 172.21.15.49:0/839972332 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7ac802ff60 0x7f7ac8032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:53.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.452+0000 7f7ae3559700 1 --2- 172.21.15.49:0/839972332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7adc0f4740 0x7f7adc1066d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:53.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.452+0000 7f7ae3559700 1 -- 172.21.15.49:0/839972332 >> 172.21.15.49:0/839972332 conn(0x7f7adc0f0260 msgr2=0x7f7adc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:53.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.452+0000 7f7ae3559700 1 -- 172.21.15.49:0/839972332 shutdown_connections 2023-11-30T21:44:53.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:53.452+0000 7f7ae3559700 1 -- 172.21.15.49:0/839972332 wait complete. 2023-11-30T21:44:53.455 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:44:53.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:53 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:51.462777+0000 mgr.smithi027.brrjcl (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:54.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:54 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:53.451638+0000 mon.smithi027 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.49:0/839972332' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:44:54.846 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:44:54.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:44:55.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:53.463161+0000 mgr.smithi027.brrjcl (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:56.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.067+0000 7f472265d700 1 -- 172.21.15.49:0/3373384759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0f6480 msgr2=0x7f471c0f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:56.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.067+0000 7f472265d700 1 --2- 172.21.15.49:0/3373384759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0f6480 0x7f471c0f68a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f470c009fb0 tx=0x7f470c009b30).stop 2023-11-30T21:44:56.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.067+0000 7f472265d700 1 -- 172.21.15.49:0/3373384759 shutdown_connections 2023-11-30T21:44:56.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.067+0000 7f472265d700 1 --2- 172.21.15.49:0/3373384759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0f6480 0x7f471c0f68a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:56.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.067+0000 7f472265d700 1 -- 172.21.15.49:0/3373384759 >> 172.21.15.49:0/3373384759 conn(0x7f471c0f1c00 msgr2=0x7f471c0f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:56.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.067+0000 7f472265d700 1 -- 172.21.15.49:0/3373384759 shutdown_connections 2023-11-30T21:44:56.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.067+0000 7f472265d700 1 -- 172.21.15.49:0/3373384759 wait complete. 2023-11-30T21:44:56.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.067+0000 7f472265d700 1 Processor -- start 2023-11-30T21:44:56.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.068+0000 7f472265d700 1 -- start start 2023-11-30T21:44:56.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.068+0000 7f472265d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0f6480 0x7f471c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:56.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.068+0000 7f472265d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c069e20 con 0x7f471c0f6480 2023-11-30T21:44:56.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.068+0000 7f471bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0f6480 0x7f471c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:56.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.068+0000 7f471bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0f6480 0x7f471c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:58414/0 (socket says 172.21.15.49:58414) 2023-11-30T21:44:56.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.068+0000 7f471bfff700 1 -- 172.21.15.49:0/691500319 learned_addr learned my addr 172.21.15.49:0/691500319 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:44:56.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.068+0000 7f471bfff700 1 -- 172.21.15.49:0/691500319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c005040 con 0x7f471c0f6480 2023-11-30T21:44:56.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.068+0000 7f471bfff700 1 --2- 172.21.15.49:0/691500319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0f6480 0x7f471c06b9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f470c013510 tx=0x7f470c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:56.072 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.069+0000 7f4718ff9700 1 -- 172.21.15.49:0/691500319 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f470c009320 con 0x7f471c0f6480 2023-11-30T21:44:56.072 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.069+0000 7f472265d700 1 -- 172.21.15.49:0/691500319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471c069fc0 con 0x7f471c0f6480 2023-11-30T21:44:56.072 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.069+0000 7f4718ff9700 1 -- 172.21.15.49:0/691500319 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f470c013f00 con 0x7f471c0f6480 2023-11-30T21:44:56.072 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.069+0000 7f4718ff9700 1 -- 172.21.15.49:0/691500319 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f470c014d90 con 0x7f471c0f6480 2023-11-30T21:44:56.072 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.069+0000 7f472265d700 1 -- 172.21.15.49:0/691500319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c06a450 con 0x7f471c0f6480 2023-11-30T21:44:56.072 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.069+0000 7f4718ff9700 1 -- 172.21.15.49:0/691500319 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f470c0263a0 con 0x7f471c0f6480 2023-11-30T21:44:56.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.070+0000 7f4718ff9700 1 --2- 172.21.15.49:0/691500319 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f470402ffb0 0x7f4704032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:44:56.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.070+0000 7f4718ff9700 1 -- 172.21.15.49:0/691500319 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f470c040ff0 con 0x7f471c0f6480 2023-11-30T21:44:56.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.070+0000 7f472265d700 1 -- 172.21.15.49:0/691500319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471c059100 con 0x7f471c0f6480 2023-11-30T21:44:56.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.070+0000 7f471b7fe700 1 --2- 172.21.15.49:0/691500319 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f470402ffb0 0x7f4704032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:44:56.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.070+0000 7f471b7fe700 1 --2- 172.21.15.49:0/691500319 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f470402ffb0 0x7f4704032470 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f47100096a0 tx=0x7f4710006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:44:56.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.075+0000 7f4718ff9700 1 -- 172.21.15.49:0/691500319 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f470c0095f0 con 0x7f471c0f6480 2023-11-30T21:44:56.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.290+0000 7f472265d700 1 -- 172.21.15.49:0/691500319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f471c0fbf40 con 0x7f471c0f6480 2023-11-30T21:44:56.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.291+0000 7f4718ff9700 1 -- 172.21.15.49:0/691500319 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f470c02b020 con 0x7f471c0f6480 2023-11-30T21:44:56.293 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:44:56.293 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:44:56.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.293+0000 7f472265d700 1 -- 172.21.15.49:0/691500319 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f470402ffb0 msgr2=0x7f4704032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:56.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.293+0000 7f472265d700 1 --2- 172.21.15.49:0/691500319 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f470402ffb0 0x7f4704032470 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f47100096a0 tx=0x7f4710006b40).stop 2023-11-30T21:44:56.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.293+0000 7f472265d700 1 -- 172.21.15.49:0/691500319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0f6480 msgr2=0x7f471c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:44:56.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.293+0000 7f472265d700 1 --2- 172.21.15.49:0/691500319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0f6480 0x7f471c06b9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f470c013510 tx=0x7f470c00b040).stop 2023-11-30T21:44:56.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.293+0000 7f472265d700 1 -- 172.21.15.49:0/691500319 shutdown_connections 2023-11-30T21:44:56.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.293+0000 7f472265d700 1 --2- 172.21.15.49:0/691500319 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f470402ffb0 0x7f4704032470 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:56.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.293+0000 7f472265d700 1 --2- 172.21.15.49:0/691500319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471c0f6480 0x7f471c06b9a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:44:56.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.293+0000 7f472265d700 1 -- 172.21.15.49:0/691500319 >> 172.21.15.49:0/691500319 conn(0x7f471c0f1c00 msgr2=0x7f471c0f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:44:56.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.293+0000 7f472265d700 1 -- 172.21.15.49:0/691500319 shutdown_connections 2023-11-30T21:44:56.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:44:56.293+0000 7f472265d700 1 -- 172.21.15.49:0/691500319 wait complete. 2023-11-30T21:44:56.296 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:44:57.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:57 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:55.463539+0000 mgr.smithi027.brrjcl (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:44:57.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:57 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:44:56.292629+0000 mon.smithi027 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.49:0/691500319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:44:57.714 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:44:57.715 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:44:59.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:44:59 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:57.463893+0000 mgr.smithi027.brrjcl (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:00.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.510+0000 7f970413c700 1 -- 172.21.15.49:0/3534293766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc0f5410 msgr2=0x7f96fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:00.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.510+0000 7f970413c700 1 --2- 172.21.15.49:0/3534293766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc0f5410 0x7f96fc0f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f96f8009fb0 tx=0x7f96f8009b30).stop 2023-11-30T21:45:00.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.510+0000 7f970413c700 1 -- 172.21.15.49:0/3534293766 shutdown_connections 2023-11-30T21:45:00.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.510+0000 7f970413c700 1 --2- 172.21.15.49:0/3534293766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc0f5410 0x7f96fc0f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:00.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.510+0000 7f970413c700 1 -- 172.21.15.49:0/3534293766 >> 172.21.15.49:0/3534293766 conn(0x7f96fc0f0b70 msgr2=0x7f96fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:00.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.510+0000 7f970413c700 1 -- 172.21.15.49:0/3534293766 shutdown_connections 2023-11-30T21:45:00.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.510+0000 7f970413c700 1 -- 172.21.15.49:0/3534293766 wait complete. 2023-11-30T21:45:00.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.510+0000 7f970413c700 1 Processor -- start 2023-11-30T21:45:00.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.510+0000 7f970413c700 1 -- start start 2023-11-30T21:45:00.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.510+0000 7f970413c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc0f5410 0x7f96fc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:00.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.511+0000 7f970413c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc105b20 con 0x7f96fc0f5410 2023-11-30T21:45:00.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.511+0000 7f9701ed8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc0f5410 0x7f96fc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:00.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.511+0000 7f9701ed8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc0f5410 0x7f96fc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:45878/0 (socket says 172.21.15.49:45878) 2023-11-30T21:45:00.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.511+0000 7f9701ed8700 1 -- 172.21.15.49:0/2602434207 learned_addr learned my addr 172.21.15.49:0/2602434207 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:00.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.511+0000 7f9701ed8700 1 -- 172.21.15.49:0/2602434207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f8005040 con 0x7f96fc0f5410 2023-11-30T21:45:00.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.511+0000 7f9701ed8700 1 --2- 172.21.15.49:0/2602434207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc0f5410 0x7f96fc105630 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f96f8014000 tx=0x7f96f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:00.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.511+0000 7f96ee7fc700 1 -- 172.21.15.49:0/2602434207 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96f80095f0 con 0x7f96fc0f5410 2023-11-30T21:45:00.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.511+0000 7f970413c700 1 -- 172.21.15.49:0/2602434207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96fc105d20 con 0x7f96fc0f5410 2023-11-30T21:45:00.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.511+0000 7f96ee7fc700 1 -- 172.21.15.49:0/2602434207 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96f8007e70 con 0x7f96fc0f5410 2023-11-30T21:45:00.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.512+0000 7f96ee7fc700 1 -- 172.21.15.49:0/2602434207 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96f8014ea0 con 0x7f96fc0f5410 2023-11-30T21:45:00.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.512+0000 7f970413c700 1 -- 172.21.15.49:0/2602434207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96fc1061b0 con 0x7f96fc0f5410 2023-11-30T21:45:00.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.512+0000 7f96ee7fc700 1 -- 172.21.15.49:0/2602434207 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96f8025070 con 0x7f96fc0f5410 2023-11-30T21:45:00.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.512+0000 7f970413c700 1 -- 172.21.15.49:0/2602434207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96fc045ad0 con 0x7f96fc0f5410 2023-11-30T21:45:00.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.512+0000 7f96ee7fc700 1 --2- 172.21.15.49:0/2602434207 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f96e802ff60 0x7f96e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:00.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.513+0000 7f96ee7fc700 1 -- 172.21.15.49:0/2602434207 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f96f8040a30 con 0x7f96fc0f5410 2023-11-30T21:45:00.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.513+0000 7f97016d7700 1 --2- 172.21.15.49:0/2602434207 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f96e802ff60 0x7f96e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:00.517 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.513+0000 7f97016d7700 1 --2- 172.21.15.49:0/2602434207 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f96e802ff60 0x7f96e8032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f96f00096a0 tx=0x7f96f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:00.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.517+0000 7f96ee7fc700 1 -- 172.21.15.49:0/2602434207 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96f80079c0 con 0x7f96fc0f5410 2023-11-30T21:45:00.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.736+0000 7f970413c700 1 -- 172.21.15.49:0/2602434207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96fc0fae60 con 0x7f96fc0f5410 2023-11-30T21:45:00.739 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.737+0000 7f96ee7fc700 1 -- 172.21.15.49:0/2602434207 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f96f800f300 con 0x7f96fc0f5410 2023-11-30T21:45:00.739 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:00.739 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:00.740 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.739+0000 7f970413c700 1 -- 172.21.15.49:0/2602434207 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f96e802ff60 msgr2=0x7f96e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:00.740 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.739+0000 7f970413c700 1 --2- 172.21.15.49:0/2602434207 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f96e802ff60 0x7f96e8032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f96f00096a0 tx=0x7f96f0006b40).stop 2023-11-30T21:45:00.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.739+0000 7f970413c700 1 -- 172.21.15.49:0/2602434207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc0f5410 msgr2=0x7f96fc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:00.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.739+0000 7f970413c700 1 --2- 172.21.15.49:0/2602434207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc0f5410 0x7f96fc105630 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f96f8014000 tx=0x7f96f800b040).stop 2023-11-30T21:45:00.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.739+0000 7f970413c700 1 -- 172.21.15.49:0/2602434207 shutdown_connections 2023-11-30T21:45:00.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.739+0000 7f970413c700 1 --2- 172.21.15.49:0/2602434207 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f96e802ff60 0x7f96e8032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:00.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.739+0000 7f970413c700 1 --2- 172.21.15.49:0/2602434207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96fc0f5410 0x7f96fc105630 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:00.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.739+0000 7f970413c700 1 -- 172.21.15.49:0/2602434207 >> 172.21.15.49:0/2602434207 conn(0x7f96fc0f0b70 msgr2=0x7f96fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:00.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.739+0000 7f970413c700 1 -- 172.21.15.49:0/2602434207 shutdown_connections 2023-11-30T21:45:00.741 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:00.739+0000 7f970413c700 1 -- 172.21.15.49:0/2602434207 wait complete. 2023-11-30T21:45:00.741 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:01.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:01 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:44:59.464121+0000 mgr.smithi027.brrjcl (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:01.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:01 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:00.738713+0000 mon.smithi027 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.49:0/2602434207' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:02.134 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:02.134 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:03.287 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.284+0000 7f5323d70700 1 -- 172.21.15.49:0/118870488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531c060280 msgr2=0x7f531c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:03.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.284+0000 7f5323d70700 1 --2- 172.21.15.49:0/118870488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531c060280 0x7f531c0f90f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f530c009fb0 tx=0x7f530c009b30).stop 2023-11-30T21:45:03.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.284+0000 7f5323d70700 1 -- 172.21.15.49:0/118870488 shutdown_connections 2023-11-30T21:45:03.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.284+0000 7f5323d70700 1 --2- 172.21.15.49:0/118870488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531c060280 0x7f531c0f90f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:03.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.284+0000 7f5323d70700 1 -- 172.21.15.49:0/118870488 >> 172.21.15.49:0/118870488 conn(0x7f531c0f0b70 msgr2=0x7f531c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:03.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.284+0000 7f5323d70700 1 -- 172.21.15.49:0/118870488 shutdown_connections 2023-11-30T21:45:03.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.284+0000 7f5323d70700 1 -- 172.21.15.49:0/118870488 wait complete. 2023-11-30T21:45:03.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.284+0000 7f5323d70700 1 Processor -- start 2023-11-30T21:45:03.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.284+0000 7f5323d70700 1 -- start start 2023-11-30T21:45:03.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.285+0000 7f5323d70700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531c060280 0x7f531c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:03.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.285+0000 7f5323d70700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c0f4ff0 con 0x7f531c060280 2023-11-30T21:45:03.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.285+0000 7f5321b0c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531c060280 0x7f531c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:03.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.285+0000 7f5321b0c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531c060280 0x7f531c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:45888/0 (socket says 172.21.15.49:45888) 2023-11-30T21:45:03.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.285+0000 7f5321b0c700 1 -- 172.21.15.49:0/3402200219 learned_addr learned my addr 172.21.15.49:0/3402200219 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:03.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.285+0000 7f5321b0c700 1 -- 172.21.15.49:0/3402200219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c005040 con 0x7f531c060280 2023-11-30T21:45:03.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.285+0000 7f5321b0c700 1 --2- 172.21.15.49:0/3402200219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531c060280 0x7f531c0f6b70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f530c013510 tx=0x7f530c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:03.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.285+0000 7f53127fc700 1 -- 172.21.15.49:0/3402200219 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f530c009320 con 0x7f531c060280 2023-11-30T21:45:03.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.285+0000 7f53127fc700 1 -- 172.21.15.49:0/3402200219 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f530c013f00 con 0x7f531c060280 2023-11-30T21:45:03.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.285+0000 7f5323d70700 1 -- 172.21.15.49:0/3402200219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531c0f5190 con 0x7f531c060280 2023-11-30T21:45:03.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.285+0000 7f53127fc700 1 -- 172.21.15.49:0/3402200219 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f530c014dc0 con 0x7f531c060280 2023-11-30T21:45:03.291 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.286+0000 7f5323d70700 1 -- 172.21.15.49:0/3402200219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531c0f5620 con 0x7f531c060280 2023-11-30T21:45:03.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.286+0000 7f53127fc700 1 -- 172.21.15.49:0/3402200219 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f530c014420 con 0x7f531c060280 2023-11-30T21:45:03.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.286+0000 7f5323d70700 1 -- 172.21.15.49:0/3402200219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f531c059100 con 0x7f531c060280 2023-11-30T21:45:03.292 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.287+0000 7f53127fc700 1 --2- 172.21.15.49:0/3402200219 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f530802ff60 0x7f5308032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:03.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.287+0000 7f53127fc700 1 -- 172.21.15.49:0/3402200219 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f530c041050 con 0x7f531c060280 2023-11-30T21:45:03.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.287+0000 7f532130b700 1 --2- 172.21.15.49:0/3402200219 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f530802ff60 0x7f5308032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:03.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.287+0000 7f532130b700 1 --2- 172.21.15.49:0/3402200219 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f530802ff60 0x7f5308032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f53180096a0 tx=0x7f5318006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:03.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.291+0000 7f53127fc700 1 -- 172.21.15.49:0/3402200219 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f530c0095f0 con 0x7f531c060280 2023-11-30T21:45:03.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:03 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:01.464361+0000 mgr.smithi027.brrjcl (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:03.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.511+0000 7f5323d70700 1 -- 172.21.15.49:0/3402200219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f531c0fbf50 con 0x7f531c060280 2023-11-30T21:45:03.514 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.513+0000 7f53127fc700 1 -- 172.21.15.49:0/3402200219 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f530c01cb10 con 0x7f531c060280 2023-11-30T21:45:03.514 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:03.514 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:03.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.513+0000 7f5323d70700 1 -- 172.21.15.49:0/3402200219 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f530802ff60 msgr2=0x7f5308032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:03.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.513+0000 7f5323d70700 1 --2- 172.21.15.49:0/3402200219 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f530802ff60 0x7f5308032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f53180096a0 tx=0x7f5318006b40).stop 2023-11-30T21:45:03.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.513+0000 7f5323d70700 1 -- 172.21.15.49:0/3402200219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531c060280 msgr2=0x7f531c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:03.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.513+0000 7f5323d70700 1 --2- 172.21.15.49:0/3402200219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531c060280 0x7f531c0f6b70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f530c013510 tx=0x7f530c00b040).stop 2023-11-30T21:45:03.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.513+0000 7f5323d70700 1 -- 172.21.15.49:0/3402200219 shutdown_connections 2023-11-30T21:45:03.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.513+0000 7f5323d70700 1 --2- 172.21.15.49:0/3402200219 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f530802ff60 0x7f5308032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:03.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.513+0000 7f5323d70700 1 --2- 172.21.15.49:0/3402200219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531c060280 0x7f531c0f6b70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:03.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.513+0000 7f5323d70700 1 -- 172.21.15.49:0/3402200219 >> 172.21.15.49:0/3402200219 conn(0x7f531c0f0b70 msgr2=0x7f531c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:03.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.513+0000 7f5323d70700 1 -- 172.21.15.49:0/3402200219 shutdown_connections 2023-11-30T21:45:03.516 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:03.513+0000 7f5323d70700 1 -- 172.21.15.49:0/3402200219 wait complete. 2023-11-30T21:45:03.516 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:04.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:03.513247+0000 mon.smithi027 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.49:0/3402200219' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:03.583061+0000 mon.smithi027 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/mirror_snapshot_schedule"}]: dispatch 2023-11-30T21:45:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:03.590590+0000 mon.smithi027 (mon.0) 126 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/trash_purge_schedule"}]: dispatch 2023-11-30T21:45:04.913 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:04.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:05.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:03.464608+0000 mgr.smithi027.brrjcl (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:06.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.808+0000 7f815ef92700 1 -- 172.21.15.49:0/963871955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81580f6510 msgr2=0x7f81580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:06.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.808+0000 7f815ef92700 1 --2- 172.21.15.49:0/963871955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81580f6510 0x7f81580f6930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f814c009fb0 tx=0x7f814c009b30).stop 2023-11-30T21:45:06.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.808+0000 7f815ef92700 1 -- 172.21.15.49:0/963871955 shutdown_connections 2023-11-30T21:45:06.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.808+0000 7f815ef92700 1 --2- 172.21.15.49:0/963871955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81580f6510 0x7f81580f6930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:06.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.808+0000 7f815ef92700 1 -- 172.21.15.49:0/963871955 >> 172.21.15.49:0/963871955 conn(0x7f81580f1c70 msgr2=0x7f81580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:06.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.808+0000 7f815ef92700 1 -- 172.21.15.49:0/963871955 shutdown_connections 2023-11-30T21:45:06.810 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.808+0000 7f815ef92700 1 -- 172.21.15.49:0/963871955 wait complete. 2023-11-30T21:45:06.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.809+0000 7f815ef92700 1 Processor -- start 2023-11-30T21:45:06.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.809+0000 7f815ef92700 1 -- start start 2023-11-30T21:45:06.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.809+0000 7f815ef92700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81580f6510 0x7f815806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:06.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.809+0000 7f815ef92700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8158069e20 con 0x7f81580f6510 2023-11-30T21:45:06.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.809+0000 7f815cd2e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81580f6510 0x7f815806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:06.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.809+0000 7f815cd2e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81580f6510 0x7f815806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:45902/0 (socket says 172.21.15.49:45902) 2023-11-30T21:45:06.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.809+0000 7f815cd2e700 1 -- 172.21.15.49:0/2817678223 learned_addr learned my addr 172.21.15.49:0/2817678223 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:06.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.810+0000 7f815cd2e700 1 -- 172.21.15.49:0/2817678223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f814c005040 con 0x7f81580f6510 2023-11-30T21:45:06.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.810+0000 7f815cd2e700 1 --2- 172.21.15.49:0/2817678223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81580f6510 0x7f815806b9a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f814c013510 tx=0x7f814c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:06.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.810+0000 7f81557fa700 1 -- 172.21.15.49:0/2817678223 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f814c009320 con 0x7f81580f6510 2023-11-30T21:45:06.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.810+0000 7f815ef92700 1 -- 172.21.15.49:0/2817678223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8158069fc0 con 0x7f81580f6510 2023-11-30T21:45:06.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.810+0000 7f81557fa700 1 -- 172.21.15.49:0/2817678223 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f814c013f00 con 0x7f81580f6510 2023-11-30T21:45:06.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.810+0000 7f81557fa700 1 -- 172.21.15.49:0/2817678223 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f814c014d90 con 0x7f81580f6510 2023-11-30T21:45:06.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.810+0000 7f815ef92700 1 -- 172.21.15.49:0/2817678223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f815806a450 con 0x7f81580f6510 2023-11-30T21:45:06.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.811+0000 7f81557fa700 1 -- 172.21.15.49:0/2817678223 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f814c014420 con 0x7f81580f6510 2023-11-30T21:45:06.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.811+0000 7f81557fa700 1 --2- 172.21.15.49:0/2817678223 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f814802ff60 0x7f8148032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:06.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.811+0000 7f815ef92700 1 -- 172.21.15.49:0/2817678223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8158059100 con 0x7f81580f6510 2023-11-30T21:45:06.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.811+0000 7f81557fa700 1 -- 172.21.15.49:0/2817678223 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f814c041000 con 0x7f81580f6510 2023-11-30T21:45:06.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.811+0000 7f8157fff700 1 --2- 172.21.15.49:0/2817678223 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f814802ff60 0x7f8148032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:06.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.811+0000 7f8157fff700 1 --2- 172.21.15.49:0/2817678223 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f814802ff60 0x7f8148032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f81440096a0 tx=0x7f8144006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:06.817 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:06.816+0000 7f81557fa700 1 -- 172.21.15.49:0/2817678223 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f814c06d2b0 con 0x7f81580f6510 2023-11-30T21:45:07.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:07.032+0000 7f815ef92700 1 -- 172.21.15.49:0/2817678223 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81580fbf60 con 0x7f81580f6510 2023-11-30T21:45:07.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:07.033+0000 7f81557fa700 1 -- 172.21.15.49:0/2817678223 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f814c01dd50 con 0x7f81580f6510 2023-11-30T21:45:07.033 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:07.033 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:07.034+0000 7f815ef92700 1 -- 172.21.15.49:0/2817678223 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f814802ff60 msgr2=0x7f8148032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:07.034+0000 7f815ef92700 1 --2- 172.21.15.49:0/2817678223 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f814802ff60 0x7f8148032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f81440096a0 tx=0x7f8144006b40).stop 2023-11-30T21:45:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:07.034+0000 7f815ef92700 1 -- 172.21.15.49:0/2817678223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81580f6510 msgr2=0x7f815806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:07.034+0000 7f815ef92700 1 --2- 172.21.15.49:0/2817678223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81580f6510 0x7f815806b9a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f814c013510 tx=0x7f814c00b040).stop 2023-11-30T21:45:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:07.034+0000 7f815ef92700 1 -- 172.21.15.49:0/2817678223 shutdown_connections 2023-11-30T21:45:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:07.034+0000 7f815ef92700 1 --2- 172.21.15.49:0/2817678223 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f814802ff60 0x7f8148032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:07.034+0000 7f815ef92700 1 --2- 172.21.15.49:0/2817678223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81580f6510 0x7f815806b9a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:07.034+0000 7f815ef92700 1 -- 172.21.15.49:0/2817678223 >> 172.21.15.49:0/2817678223 conn(0x7f81580f1c70 msgr2=0x7f81580f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:07.034+0000 7f815ef92700 1 -- 172.21.15.49:0/2817678223 shutdown_connections 2023-11-30T21:45:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:07.034+0000 7f815ef92700 1 -- 172.21.15.49:0/2817678223 wait complete. 2023-11-30T21:45:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:07.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:07 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:05.464948+0000 mgr.smithi027.brrjcl (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:07.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:07 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:07.033378+0000 mon.smithi027 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.49:0/2817678223' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:08.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:08.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:09.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:09 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:07.465265+0000 mgr.smithi027.brrjcl (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:09.659 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.658+0000 7f4c65df8700 1 -- 172.21.15.49:0/577320453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c600f5410 msgr2=0x7f4c600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:09.659 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.658+0000 7f4c65df8700 1 --2- 172.21.15.49:0/577320453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c600f5410 0x7f4c600f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f4c50009fb0 tx=0x7f4c50009b30).stop 2023-11-30T21:45:09.659 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.658+0000 7f4c65df8700 1 -- 172.21.15.49:0/577320453 shutdown_connections 2023-11-30T21:45:09.659 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.658+0000 7f4c65df8700 1 --2- 172.21.15.49:0/577320453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c600f5410 0x7f4c600f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:09.659 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.658+0000 7f4c65df8700 1 -- 172.21.15.49:0/577320453 >> 172.21.15.49:0/577320453 conn(0x7f4c600f0b70 msgr2=0x7f4c600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:09.659 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.658+0000 7f4c65df8700 1 -- 172.21.15.49:0/577320453 shutdown_connections 2023-11-30T21:45:09.659 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.658+0000 7f4c65df8700 1 -- 172.21.15.49:0/577320453 wait complete. 2023-11-30T21:45:09.660 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.659+0000 7f4c65df8700 1 Processor -- start 2023-11-30T21:45:09.660 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.659+0000 7f4c65df8700 1 -- start start 2023-11-30T21:45:09.660 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.659+0000 7f4c65df8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c600f5410 0x7f4c60109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:09.660 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.659+0000 7f4c65df8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c60060220 con 0x7f4c600f5410 2023-11-30T21:45:09.660 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.660+0000 7f4c5f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c600f5410 0x7f4c60109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:09.660 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.660+0000 7f4c5f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c600f5410 0x7f4c60109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59962/0 (socket says 172.21.15.49:59962) 2023-11-30T21:45:09.661 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.660+0000 7f4c5f7fe700 1 -- 172.21.15.49:0/2979927537 learned_addr learned my addr 172.21.15.49:0/2979927537 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:09.661 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.660+0000 7f4c5f7fe700 1 -- 172.21.15.49:0/2979927537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c50005040 con 0x7f4c600f5410 2023-11-30T21:45:09.661 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.660+0000 7f4c5f7fe700 1 --2- 172.21.15.49:0/2979927537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c600f5410 0x7f4c60109450 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f4c50013510 tx=0x7f4c5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:09.662 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.660+0000 7f4c57fff700 1 -- 172.21.15.49:0/2979927537 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c50009320 con 0x7f4c600f5410 2023-11-30T21:45:09.662 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.660+0000 7f4c65df8700 1 -- 172.21.15.49:0/2979927537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c60060420 con 0x7f4c600f5410 2023-11-30T21:45:09.662 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.660+0000 7f4c57fff700 1 -- 172.21.15.49:0/2979927537 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c50013f00 con 0x7f4c600f5410 2023-11-30T21:45:09.662 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.661+0000 7f4c57fff700 1 -- 172.21.15.49:0/2979927537 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c50014d90 con 0x7f4c600f5410 2023-11-30T21:45:09.662 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.661+0000 7f4c65df8700 1 -- 172.21.15.49:0/2979927537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c60109b00 con 0x7f4c600f5410 2023-11-30T21:45:09.663 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.662+0000 7f4c57fff700 1 -- 172.21.15.49:0/2979927537 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c500095f0 con 0x7f4c600f5410 2023-11-30T21:45:09.663 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.662+0000 7f4c65df8700 1 -- 172.21.15.49:0/2979927537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c60059100 con 0x7f4c600f5410 2023-11-30T21:45:09.663 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.662+0000 7f4c57fff700 1 --2- 172.21.15.49:0/2979927537 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4c4c02ff60 0x7f4c4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:09.663 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.662+0000 7f4c57fff700 1 -- 172.21.15.49:0/2979927537 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c50040fc0 con 0x7f4c600f5410 2023-11-30T21:45:09.663 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.662+0000 7f4c5effd700 1 --2- 172.21.15.49:0/2979927537 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4c4c02ff60 0x7f4c4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:09.663 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.662+0000 7f4c5effd700 1 --2- 172.21.15.49:0/2979927537 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4c4c02ff60 0x7f4c4c032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f4c480096a0 tx=0x7f4c48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:09.668 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.667+0000 7f4c57fff700 1 -- 172.21.15.49:0/2979927537 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c50014420 con 0x7f4c600f5410 2023-11-30T21:45:09.886 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.885+0000 7f4c65df8700 1 -- 172.21.15.49:0/2979927537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c60045ad0 con 0x7f4c600f5410 2023-11-30T21:45:09.886 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.886+0000 7f4c57fff700 1 -- 172.21.15.49:0/2979927537 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c50010230 con 0x7f4c600f5410 2023-11-30T21:45:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:09.887 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:09.888 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.887+0000 7f4c65df8700 1 -- 172.21.15.49:0/2979927537 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4c4c02ff60 msgr2=0x7f4c4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:09.888 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.887+0000 7f4c65df8700 1 --2- 172.21.15.49:0/2979927537 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4c4c02ff60 0x7f4c4c032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f4c480096a0 tx=0x7f4c48006b40).stop 2023-11-30T21:45:09.888 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.887+0000 7f4c65df8700 1 -- 172.21.15.49:0/2979927537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c600f5410 msgr2=0x7f4c60109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:09.888 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.887+0000 7f4c65df8700 1 --2- 172.21.15.49:0/2979927537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c600f5410 0x7f4c60109450 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f4c50013510 tx=0x7f4c5000b040).stop 2023-11-30T21:45:09.888 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.888+0000 7f4c65df8700 1 -- 172.21.15.49:0/2979927537 shutdown_connections 2023-11-30T21:45:09.888 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.888+0000 7f4c65df8700 1 --2- 172.21.15.49:0/2979927537 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4c4c02ff60 0x7f4c4c032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:09.888 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.888+0000 7f4c65df8700 1 --2- 172.21.15.49:0/2979927537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c600f5410 0x7f4c60109450 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:09.889 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.888+0000 7f4c65df8700 1 -- 172.21.15.49:0/2979927537 >> 172.21.15.49:0/2979927537 conn(0x7f4c600f0b70 msgr2=0x7f4c600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:09.889 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.888+0000 7f4c65df8700 1 -- 172.21.15.49:0/2979927537 shutdown_connections 2023-11-30T21:45:09.889 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:09.888+0000 7f4c65df8700 1 -- 172.21.15.49:0/2979927537 wait complete. 2023-11-30T21:45:09.889 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:10.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:10 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:09.886351+0000 mon.smithi027 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.49:0/2979927537' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:11.237 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:11.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:11.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:09.465530+0000 mgr.smithi027.brrjcl (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:12.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.836+0000 7fa8d4494700 1 -- 172.21.15.49:0/4023731658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8cc0f6510 msgr2=0x7fa8cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:12.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.836+0000 7fa8d4494700 1 --2- 172.21.15.49:0/4023731658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8cc0f6510 0x7fa8cc0f6930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fa8bc009fb0 tx=0x7fa8bc009b30).stop 2023-11-30T21:45:12.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.836+0000 7fa8d4494700 1 -- 172.21.15.49:0/4023731658 shutdown_connections 2023-11-30T21:45:12.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.836+0000 7fa8d4494700 1 --2- 172.21.15.49:0/4023731658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8cc0f6510 0x7fa8cc0f6930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:12.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.836+0000 7fa8d4494700 1 -- 172.21.15.49:0/4023731658 >> 172.21.15.49:0/4023731658 conn(0x7fa8cc0f1c70 msgr2=0x7fa8cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:12.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.836+0000 7fa8d4494700 1 -- 172.21.15.49:0/4023731658 shutdown_connections 2023-11-30T21:45:12.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.836+0000 7fa8d4494700 1 -- 172.21.15.49:0/4023731658 wait complete. 2023-11-30T21:45:12.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.837+0000 7fa8d4494700 1 Processor -- start 2023-11-30T21:45:12.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.837+0000 7fa8d4494700 1 -- start start 2023-11-30T21:45:12.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.837+0000 7fa8d4494700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8cc0f6510 0x7fa8cc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:12.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.837+0000 7fa8d4494700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc106c20 con 0x7fa8cc0f6510 2023-11-30T21:45:12.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.837+0000 7fa8d2230700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8cc0f6510 0x7fa8cc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:12.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.837+0000 7fa8d2230700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8cc0f6510 0x7fa8cc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59990/0 (socket says 172.21.15.49:59990) 2023-11-30T21:45:12.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.837+0000 7fa8d2230700 1 -- 172.21.15.49:0/1818970101 learned_addr learned my addr 172.21.15.49:0/1818970101 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:12.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.837+0000 7fa8d2230700 1 -- 172.21.15.49:0/1818970101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc005040 con 0x7fa8cc0f6510 2023-11-30T21:45:12.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.838+0000 7fa8d2230700 1 --2- 172.21.15.49:0/1818970101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8cc0f6510 0x7fa8cc106730 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa8bc014000 tx=0x7fa8bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:12.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.838+0000 7fa8c2ffd700 1 -- 172.21.15.49:0/1818970101 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8bc0095f0 con 0x7fa8cc0f6510 2023-11-30T21:45:12.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.838+0000 7fa8c2ffd700 1 -- 172.21.15.49:0/1818970101 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8bc007e70 con 0x7fa8cc0f6510 2023-11-30T21:45:12.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.838+0000 7fa8d4494700 1 -- 172.21.15.49:0/1818970101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8cc106e20 con 0x7fa8cc0f6510 2023-11-30T21:45:12.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.838+0000 7fa8c2ffd700 1 -- 172.21.15.49:0/1818970101 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8bc014ea0 con 0x7fa8cc0f6510 2023-11-30T21:45:12.841 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.838+0000 7fa8d4494700 1 -- 172.21.15.49:0/1818970101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8cc1072b0 con 0x7fa8cc0f6510 2023-11-30T21:45:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.839+0000 7fa8c2ffd700 1 -- 172.21.15.49:0/1818970101 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8bc025070 con 0x7fa8cc0f6510 2023-11-30T21:45:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.839+0000 7fa8d4494700 1 -- 172.21.15.49:0/1818970101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8cc045ad0 con 0x7fa8cc0f6510 2023-11-30T21:45:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.839+0000 7fa8c2ffd700 1 --2- 172.21.15.49:0/1818970101 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa8b802ff60 0x7fa8b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.839+0000 7fa8c2ffd700 1 -- 172.21.15.49:0/1818970101 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8bc040a30 con 0x7fa8cc0f6510 2023-11-30T21:45:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.839+0000 7fa8d1a2f700 1 --2- 172.21.15.49:0/1818970101 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa8b802ff60 0x7fa8b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.839+0000 7fa8d1a2f700 1 --2- 172.21.15.49:0/1818970101 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa8b802ff60 0x7fa8b8032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fa8c80096a0 tx=0x7fa8c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:12.844 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:12.844+0000 7fa8c2ffd700 1 -- 172.21.15.49:0/1818970101 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8bc0079c0 con 0x7fa8cc0f6510 2023-11-30T21:45:13.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:13.062+0000 7fa8d4494700 1 -- 172.21.15.49:0/1818970101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8cc0fbf60 con 0x7fa8cc0f6510 2023-11-30T21:45:13.064 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:13.063+0000 7fa8c2ffd700 1 -- 172.21.15.49:0/1818970101 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa8bc00f300 con 0x7fa8cc0f6510 2023-11-30T21:45:13.064 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:13.064 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:13.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:13.065+0000 7fa8d4494700 1 -- 172.21.15.49:0/1818970101 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa8b802ff60 msgr2=0x7fa8b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:13.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:13.065+0000 7fa8d4494700 1 --2- 172.21.15.49:0/1818970101 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa8b802ff60 0x7fa8b8032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fa8c80096a0 tx=0x7fa8c8006b40).stop 2023-11-30T21:45:13.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:13.065+0000 7fa8d4494700 1 -- 172.21.15.49:0/1818970101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8cc0f6510 msgr2=0x7fa8cc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:13.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:13.065+0000 7fa8d4494700 1 --2- 172.21.15.49:0/1818970101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8cc0f6510 0x7fa8cc106730 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa8bc014000 tx=0x7fa8bc00b040).stop 2023-11-30T21:45:13.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:13.065+0000 7fa8d4494700 1 -- 172.21.15.49:0/1818970101 shutdown_connections 2023-11-30T21:45:13.065 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:13.065+0000 7fa8d4494700 1 --2- 172.21.15.49:0/1818970101 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa8b802ff60 0x7fa8b8032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:13.066 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:13.065+0000 7fa8d4494700 1 --2- 172.21.15.49:0/1818970101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8cc0f6510 0x7fa8cc106730 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:13.066 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:13.065+0000 7fa8d4494700 1 -- 172.21.15.49:0/1818970101 >> 172.21.15.49:0/1818970101 conn(0x7fa8cc0f1c70 msgr2=0x7fa8cc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:13.066 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:13.065+0000 7fa8d4494700 1 -- 172.21.15.49:0/1818970101 shutdown_connections 2023-11-30T21:45:13.066 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:13.065+0000 7fa8d4494700 1 -- 172.21.15.49:0/1818970101 wait complete. 2023-11-30T21:45:13.067 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:13.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:13 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:11.465862+0000 mgr.smithi027.brrjcl (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:13.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:13 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:13.063364+0000 mon.smithi027 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.49:0/1818970101' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:14.475 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:14.476 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:15.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:15 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:13.466132+0000 mgr.smithi027.brrjcl (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:15.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.687+0000 7f9502994700 1 -- 172.21.15.49:0/3213517637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94fc0f4920 msgr2=0x7f94fc0f4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:15.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.687+0000 7f9502994700 1 --2- 172.21.15.49:0/3213517637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94fc0f4920 0x7f94fc0f4d40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f94f8009fb0 tx=0x7f94f8009b30).stop 2023-11-30T21:45:15.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.687+0000 7f9502994700 1 -- 172.21.15.49:0/3213517637 shutdown_connections 2023-11-30T21:45:15.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.687+0000 7f9502994700 1 --2- 172.21.15.49:0/3213517637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94fc0f4920 0x7f94fc0f4d40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:15.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.687+0000 7f9502994700 1 -- 172.21.15.49:0/3213517637 >> 172.21.15.49:0/3213517637 conn(0x7f94fc0f0120 msgr2=0x7f94fc0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:15.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.688+0000 7f9502994700 1 -- 172.21.15.49:0/3213517637 shutdown_connections 2023-11-30T21:45:15.689 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.688+0000 7f9502994700 1 -- 172.21.15.49:0/3213517637 wait complete. 2023-11-30T21:45:15.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.688+0000 7f9502994700 1 Processor -- start 2023-11-30T21:45:15.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.689+0000 7f9502994700 1 -- start start 2023-11-30T21:45:15.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.689+0000 7f9502994700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94fc0f4920 0x7f94fc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:15.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.689+0000 7f9502994700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc0f1700 con 0x7f94fc0f4920 2023-11-30T21:45:15.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.689+0000 7f9501992700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94fc0f4920 0x7f94fc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:15.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.689+0000 7f9501992700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94fc0f4920 0x7f94fc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59998/0 (socket says 172.21.15.49:59998) 2023-11-30T21:45:15.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.689+0000 7f9501992700 1 -- 172.21.15.49:0/383182993 learned_addr learned my addr 172.21.15.49:0/383182993 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:15.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.689+0000 7f9501992700 1 -- 172.21.15.49:0/383182993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f8005040 con 0x7f94fc0f4920 2023-11-30T21:45:15.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.689+0000 7f9501992700 1 --2- 172.21.15.49:0/383182993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94fc0f4920 0x7f94fc1091b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f94f8013510 tx=0x7f94f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:15.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.690+0000 7f94f27fc700 1 -- 172.21.15.49:0/383182993 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94f8009320 con 0x7f94fc0f4920 2023-11-30T21:45:15.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.690+0000 7f9502994700 1 -- 172.21.15.49:0/383182993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94fc0f1900 con 0x7f94fc0f4920 2023-11-30T21:45:15.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.690+0000 7f94f27fc700 1 -- 172.21.15.49:0/383182993 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94f8013f00 con 0x7f94fc0f4920 2023-11-30T21:45:15.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.690+0000 7f94f27fc700 1 -- 172.21.15.49:0/383182993 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94f8014dc0 con 0x7f94fc0f4920 2023-11-30T21:45:15.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.690+0000 7f9502994700 1 -- 172.21.15.49:0/383182993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94fc109860 con 0x7f94fc0f4920 2023-11-30T21:45:15.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.690+0000 7f94f27fc700 1 -- 172.21.15.49:0/383182993 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94f80095f0 con 0x7f94fc0f4920 2023-11-30T21:45:15.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.691+0000 7f9502994700 1 -- 172.21.15.49:0/383182993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94fc045ad0 con 0x7f94fc0f4920 2023-11-30T21:45:15.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.691+0000 7f94f27fc700 1 --2- 172.21.15.49:0/383182993 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f94e802ff60 0x7f94e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:15.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.691+0000 7f94f27fc700 1 -- 172.21.15.49:0/383182993 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94f8041080 con 0x7f94fc0f4920 2023-11-30T21:45:15.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.691+0000 7f9501191700 1 --2- 172.21.15.49:0/383182993 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f94e802ff60 0x7f94e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:15.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.691+0000 7f9501191700 1 --2- 172.21.15.49:0/383182993 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f94e802ff60 0x7f94e8032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f94ec0096a0 tx=0x7f94ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:15.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.696+0000 7f94f27fc700 1 -- 172.21.15.49:0/383182993 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94f801c790 con 0x7f94fc0f4920 2023-11-30T21:45:15.911 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.911+0000 7f9502994700 1 -- 172.21.15.49:0/383182993 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94fc027a00 con 0x7f94fc0f4920 2023-11-30T21:45:15.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.911+0000 7f94f27fc700 1 -- 172.21.15.49:0/383182993 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f94f8010230 con 0x7f94fc0f4920 2023-11-30T21:45:15.912 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:15.912 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:15.913 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.913+0000 7f9502994700 1 -- 172.21.15.49:0/383182993 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f94e802ff60 msgr2=0x7f94e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:15.913 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.913+0000 7f9502994700 1 --2- 172.21.15.49:0/383182993 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f94e802ff60 0x7f94e8032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f94ec0096a0 tx=0x7f94ec006b40).stop 2023-11-30T21:45:15.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.913+0000 7f9502994700 1 -- 172.21.15.49:0/383182993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94fc0f4920 msgr2=0x7f94fc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:15.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.913+0000 7f9502994700 1 --2- 172.21.15.49:0/383182993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94fc0f4920 0x7f94fc1091b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f94f8013510 tx=0x7f94f800b040).stop 2023-11-30T21:45:15.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.913+0000 7f9502994700 1 -- 172.21.15.49:0/383182993 shutdown_connections 2023-11-30T21:45:15.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.913+0000 7f9502994700 1 --2- 172.21.15.49:0/383182993 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f94e802ff60 0x7f94e8032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:15.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.913+0000 7f9502994700 1 --2- 172.21.15.49:0/383182993 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94fc0f4920 0x7f94fc1091b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:15.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.913+0000 7f9502994700 1 -- 172.21.15.49:0/383182993 >> 172.21.15.49:0/383182993 conn(0x7f94fc0f0120 msgr2=0x7f94fc0f2380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:15.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.913+0000 7f9502994700 1 -- 172.21.15.49:0/383182993 shutdown_connections 2023-11-30T21:45:15.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:15.913+0000 7f9502994700 1 -- 172.21.15.49:0/383182993 wait complete. 2023-11-30T21:45:15.914 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:16.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:16 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:15.911992+0000 mon.smithi027 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.49:0/383182993' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:17.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:17.343 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:17.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:15.466419+0000 mgr.smithi027.brrjcl (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:18.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.499+0000 7fb2b18f5700 1 -- 172.21.15.49:0/716312909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2ac0f5410 msgr2=0x7fb2ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:18.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.499+0000 7fb2b18f5700 1 --2- 172.21.15.49:0/716312909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2ac0f5410 0x7fb2ac0f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fb29c009fb0 tx=0x7fb29c009b30).stop 2023-11-30T21:45:18.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.499+0000 7fb2b18f5700 1 -- 172.21.15.49:0/716312909 shutdown_connections 2023-11-30T21:45:18.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.499+0000 7fb2b18f5700 1 --2- 172.21.15.49:0/716312909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2ac0f5410 0x7fb2ac0f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:18.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.499+0000 7fb2b18f5700 1 -- 172.21.15.49:0/716312909 >> 172.21.15.49:0/716312909 conn(0x7fb2ac0f0b70 msgr2=0x7fb2ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:18.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.499+0000 7fb2b18f5700 1 -- 172.21.15.49:0/716312909 shutdown_connections 2023-11-30T21:45:18.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.499+0000 7fb2b18f5700 1 -- 172.21.15.49:0/716312909 wait complete. 2023-11-30T21:45:18.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.500+0000 7fb2b18f5700 1 Processor -- start 2023-11-30T21:45:18.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.500+0000 7fb2b18f5700 1 -- start start 2023-11-30T21:45:18.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.500+0000 7fb2b18f5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2ac0f5410 0x7fb2ac109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:18.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.500+0000 7fb2b18f5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac060220 con 0x7fb2ac0f5410 2023-11-30T21:45:18.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.500+0000 7fb2aaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2ac0f5410 0x7fb2ac109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:18.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.500+0000 7fb2aaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2ac0f5410 0x7fb2ac109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:37950/0 (socket says 172.21.15.49:37950) 2023-11-30T21:45:18.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.500+0000 7fb2aaffd700 1 -- 172.21.15.49:0/1903091158 learned_addr learned my addr 172.21.15.49:0/1903091158 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:18.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.501+0000 7fb2aaffd700 1 -- 172.21.15.49:0/1903091158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c005040 con 0x7fb2ac0f5410 2023-11-30T21:45:18.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.501+0000 7fb2aaffd700 1 --2- 172.21.15.49:0/1903091158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2ac0f5410 0x7fb2ac109450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fb29c013510 tx=0x7fb29c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:18.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.501+0000 7fb2a37fe700 1 -- 172.21.15.49:0/1903091158 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb29c009320 con 0x7fb2ac0f5410 2023-11-30T21:45:18.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.501+0000 7fb2b18f5700 1 -- 172.21.15.49:0/1903091158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ac060420 con 0x7fb2ac0f5410 2023-11-30T21:45:18.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.501+0000 7fb2a37fe700 1 -- 172.21.15.49:0/1903091158 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb29c013f00 con 0x7fb2ac0f5410 2023-11-30T21:45:18.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.501+0000 7fb2a37fe700 1 -- 172.21.15.49:0/1903091158 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb29c014d90 con 0x7fb2ac0f5410 2023-11-30T21:45:18.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.501+0000 7fb2b18f5700 1 -- 172.21.15.49:0/1903091158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ac109b00 con 0x7fb2ac0f5410 2023-11-30T21:45:18.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.502+0000 7fb2a37fe700 1 -- 172.21.15.49:0/1903091158 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb29c014420 con 0x7fb2ac0f5410 2023-11-30T21:45:18.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.502+0000 7fb2b18f5700 1 -- 172.21.15.49:0/1903091158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2ac059100 con 0x7fb2ac0f5410 2023-11-30T21:45:18.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.502+0000 7fb2a37fe700 1 --2- 172.21.15.49:0/1903091158 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb29802ff60 0x7fb298032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:18.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.502+0000 7fb2a37fe700 1 -- 172.21.15.49:0/1903091158 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb29c0410c0 con 0x7fb2ac0f5410 2023-11-30T21:45:18.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.502+0000 7fb2aa7fc700 1 --2- 172.21.15.49:0/1903091158 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb29802ff60 0x7fb298032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:18.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.503+0000 7fb2aa7fc700 1 --2- 172.21.15.49:0/1903091158 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb29802ff60 0x7fb298032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fb2940096a0 tx=0x7fb294006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:18.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.507+0000 7fb2a37fe700 1 -- 172.21.15.49:0/1903091158 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb29c0095f0 con 0x7fb2ac0f5410 2023-11-30T21:45:18.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.724+0000 7fb2b18f5700 1 -- 172.21.15.49:0/1903091158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2ac045ad0 con 0x7fb2ac0f5410 2023-11-30T21:45:18.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.725+0000 7fb2a37fe700 1 -- 172.21.15.49:0/1903091158 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb29c01db70 con 0x7fb2ac0f5410 2023-11-30T21:45:18.726 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:18.726 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:18.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.726+0000 7fb2b18f5700 1 -- 172.21.15.49:0/1903091158 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb29802ff60 msgr2=0x7fb298032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:18.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.726+0000 7fb2b18f5700 1 --2- 172.21.15.49:0/1903091158 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb29802ff60 0x7fb298032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fb2940096a0 tx=0x7fb294006b40).stop 2023-11-30T21:45:18.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.726+0000 7fb2b18f5700 1 -- 172.21.15.49:0/1903091158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2ac0f5410 msgr2=0x7fb2ac109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:18.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.726+0000 7fb2b18f5700 1 --2- 172.21.15.49:0/1903091158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2ac0f5410 0x7fb2ac109450 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fb29c013510 tx=0x7fb29c00b040).stop 2023-11-30T21:45:18.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.726+0000 7fb2b18f5700 1 -- 172.21.15.49:0/1903091158 shutdown_connections 2023-11-30T21:45:18.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.726+0000 7fb2b18f5700 1 --2- 172.21.15.49:0/1903091158 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb29802ff60 0x7fb298032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:18.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.726+0000 7fb2b18f5700 1 --2- 172.21.15.49:0/1903091158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2ac0f5410 0x7fb2ac109450 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:18.730 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.726+0000 7fb2b18f5700 1 -- 172.21.15.49:0/1903091158 >> 172.21.15.49:0/1903091158 conn(0x7fb2ac0f0b70 msgr2=0x7fb2ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:18.730 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.726+0000 7fb2b18f5700 1 -- 172.21.15.49:0/1903091158 shutdown_connections 2023-11-30T21:45:18.730 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:18.726+0000 7fb2b18f5700 1 -- 172.21.15.49:0/1903091158 wait complete. 2023-11-30T21:45:18.730 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:19.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:17.466755+0000 mgr.smithi027.brrjcl (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:19.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:18.726119+0000 mon.smithi027 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.49:0/1903091158' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:20.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:20.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:21.345 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.339+0000 7fcdaf189700 1 -- 172.21.15.49:0/3013550548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcda80f5410 msgr2=0x7fcda80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:21.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.339+0000 7fcdaf189700 1 --2- 172.21.15.49:0/3013550548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcda80f5410 0x7fcda80f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fcd9c009fb0 tx=0x7fcd9c009b30).stop 2023-11-30T21:45:21.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.339+0000 7fcdaf189700 1 -- 172.21.15.49:0/3013550548 shutdown_connections 2023-11-30T21:45:21.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.339+0000 7fcdaf189700 1 --2- 172.21.15.49:0/3013550548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcda80f5410 0x7fcda80f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:21.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.339+0000 7fcdaf189700 1 -- 172.21.15.49:0/3013550548 >> 172.21.15.49:0/3013550548 conn(0x7fcda80f0b70 msgr2=0x7fcda80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:21.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.339+0000 7fcdaf189700 1 -- 172.21.15.49:0/3013550548 shutdown_connections 2023-11-30T21:45:21.347 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.340+0000 7fcdaf189700 1 -- 172.21.15.49:0/3013550548 wait complete. 2023-11-30T21:45:21.347 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.340+0000 7fcdaf189700 1 Processor -- start 2023-11-30T21:45:21.347 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.340+0000 7fcdaf189700 1 -- start start 2023-11-30T21:45:21.347 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.341+0000 7fcdaf189700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcda80f5410 0x7fcda80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:21.347 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.341+0000 7fcdaf189700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda80fb820 con 0x7fcda80f5410 2023-11-30T21:45:21.348 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.341+0000 7fcdacf25700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcda80f5410 0x7fcda80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:21.349 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.341+0000 7fcdacf25700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcda80f5410 0x7fcda80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:37968/0 (socket says 172.21.15.49:37968) 2023-11-30T21:45:21.349 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.341+0000 7fcdacf25700 1 -- 172.21.15.49:0/3089110376 learned_addr learned my addr 172.21.15.49:0/3089110376 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:21.349 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.341+0000 7fcdacf25700 1 -- 172.21.15.49:0/3089110376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c005040 con 0x7fcda80f5410 2023-11-30T21:45:21.349 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.342+0000 7fcdacf25700 1 --2- 172.21.15.49:0/3089110376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcda80f5410 0x7fcda80fd3a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fcd9c013510 tx=0x7fcd9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:21.349 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.342+0000 7fcda57fa700 1 -- 172.21.15.49:0/3089110376 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd9c009320 con 0x7fcda80f5410 2023-11-30T21:45:21.349 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.342+0000 7fcdaf189700 1 -- 172.21.15.49:0/3089110376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda80fb9c0 con 0x7fcda80f5410 2023-11-30T21:45:21.349 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.342+0000 7fcda57fa700 1 -- 172.21.15.49:0/3089110376 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd9c013f00 con 0x7fcda80f5410 2023-11-30T21:45:21.349 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.342+0000 7fcda57fa700 1 -- 172.21.15.49:0/3089110376 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd9c014d90 con 0x7fcda80f5410 2023-11-30T21:45:21.349 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.342+0000 7fcdaf189700 1 -- 172.21.15.49:0/3089110376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda80fbe50 con 0x7fcda80f5410 2023-11-30T21:45:21.349 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.343+0000 7fcda57fa700 1 -- 172.21.15.49:0/3089110376 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd9c014420 con 0x7fcda80f5410 2023-11-30T21:45:21.350 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.343+0000 7fcda57fa700 1 --2- 172.21.15.49:0/3089110376 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcd9802ff60 0x7fcd98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:21.350 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.343+0000 7fcdaf189700 1 -- 172.21.15.49:0/3089110376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda8059100 con 0x7fcda80f5410 2023-11-30T21:45:21.350 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.343+0000 7fcda7fff700 1 --2- 172.21.15.49:0/3089110376 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcd9802ff60 0x7fcd98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:21.350 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.343+0000 7fcda57fa700 1 -- 172.21.15.49:0/3089110376 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd9c040ff0 con 0x7fcda80f5410 2023-11-30T21:45:21.350 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.343+0000 7fcda7fff700 1 --2- 172.21.15.49:0/3089110376 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcd9802ff60 0x7fcd98032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fcd940096a0 tx=0x7fcd94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:21.351 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.348+0000 7fcda57fa700 1 -- 172.21.15.49:0/3089110376 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd9c0095f0 con 0x7fcda80f5410 2023-11-30T21:45:21.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:21 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:19.467103+0000 mgr.smithi027.brrjcl (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:21.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.566+0000 7fcdaf189700 1 -- 172.21.15.49:0/3089110376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcda80fae60 con 0x7fcda80f5410 2023-11-30T21:45:21.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.567+0000 7fcda57fa700 1 -- 172.21.15.49:0/3089110376 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcd9c01db70 con 0x7fcda80f5410 2023-11-30T21:45:21.568 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:21.569 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:21.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.568+0000 7fcdaf189700 1 -- 172.21.15.49:0/3089110376 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcd9802ff60 msgr2=0x7fcd98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:21.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.568+0000 7fcdaf189700 1 --2- 172.21.15.49:0/3089110376 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcd9802ff60 0x7fcd98032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fcd940096a0 tx=0x7fcd94006b40).stop 2023-11-30T21:45:21.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.568+0000 7fcdaf189700 1 -- 172.21.15.49:0/3089110376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcda80f5410 msgr2=0x7fcda80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:21.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.568+0000 7fcdaf189700 1 --2- 172.21.15.49:0/3089110376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcda80f5410 0x7fcda80fd3a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fcd9c013510 tx=0x7fcd9c00b040).stop 2023-11-30T21:45:21.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.568+0000 7fcdaf189700 1 -- 172.21.15.49:0/3089110376 shutdown_connections 2023-11-30T21:45:21.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.568+0000 7fcdaf189700 1 --2- 172.21.15.49:0/3089110376 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcd9802ff60 0x7fcd98032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:21.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.568+0000 7fcdaf189700 1 --2- 172.21.15.49:0/3089110376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcda80f5410 0x7fcda80fd3a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:21.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.568+0000 7fcdaf189700 1 -- 172.21.15.49:0/3089110376 >> 172.21.15.49:0/3089110376 conn(0x7fcda80f0b70 msgr2=0x7fcda80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:21.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.568+0000 7fcdaf189700 1 -- 172.21.15.49:0/3089110376 shutdown_connections 2023-11-30T21:45:21.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:21.568+0000 7fcdaf189700 1 -- 172.21.15.49:0/3089110376 wait complete. 2023-11-30T21:45:21.571 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:22.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:22 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:21.568234+0000 mon.smithi027 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.49:0/3089110376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:22.965 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:22.965 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:23.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:23 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:21.467463+0000 mgr.smithi027.brrjcl (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:24.203 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.201+0000 7f715d39f700 1 -- 172.21.15.49:0/1589654548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71580f78e0 msgr2=0x7f71580f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:24.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.201+0000 7f715d39f700 1 --2- 172.21.15.49:0/1589654548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71580f78e0 0x7f71580f7d00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f7148009fb0 tx=0x7f7148009b30).stop 2023-11-30T21:45:24.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.201+0000 7f715d39f700 1 -- 172.21.15.49:0/1589654548 shutdown_connections 2023-11-30T21:45:24.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.201+0000 7f715d39f700 1 --2- 172.21.15.49:0/1589654548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71580f78e0 0x7f71580f7d00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:24.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.201+0000 7f715d39f700 1 -- 172.21.15.49:0/1589654548 >> 172.21.15.49:0/1589654548 conn(0x7f71580f0120 msgr2=0x7f71580f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:24.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.202+0000 7f715d39f700 1 -- 172.21.15.49:0/1589654548 shutdown_connections 2023-11-30T21:45:24.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.202+0000 7f715d39f700 1 -- 172.21.15.49:0/1589654548 wait complete. 2023-11-30T21:45:24.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.202+0000 7f715d39f700 1 Processor -- start 2023-11-30T21:45:24.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.202+0000 7f715d39f700 1 -- start start 2023-11-30T21:45:24.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.203+0000 7f715d39f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71580f78e0 0x7f71581064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:24.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.203+0000 7f715d39f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71581069d0 con 0x7f71580f78e0 2023-11-30T21:45:24.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.203+0000 7f7157fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71580f78e0 0x7f71581064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:24.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.203+0000 7f7157fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71580f78e0 0x7f71581064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:37978/0 (socket says 172.21.15.49:37978) 2023-11-30T21:45:24.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.203+0000 7f7157fff700 1 -- 172.21.15.49:0/2929783360 learned_addr learned my addr 172.21.15.49:0/2929783360 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:24.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.203+0000 7f7157fff700 1 -- 172.21.15.49:0/2929783360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7148005040 con 0x7f71580f78e0 2023-11-30T21:45:24.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.204+0000 7f7157fff700 1 --2- 172.21.15.49:0/2929783360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71580f78e0 0x7f71581064e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f7148013510 tx=0x7f714800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:24.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.204+0000 7f7154ff9700 1 -- 172.21.15.49:0/2929783360 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7148009320 con 0x7f71580f78e0 2023-11-30T21:45:24.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.204+0000 7f715d39f700 1 -- 172.21.15.49:0/2929783360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7158106bd0 con 0x7f71580f78e0 2023-11-30T21:45:24.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.204+0000 7f7154ff9700 1 -- 172.21.15.49:0/2929783360 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7148013f00 con 0x7f71580f78e0 2023-11-30T21:45:24.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.204+0000 7f7154ff9700 1 -- 172.21.15.49:0/2929783360 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7148014dc0 con 0x7f71580f78e0 2023-11-30T21:45:24.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.204+0000 7f715d39f700 1 -- 172.21.15.49:0/2929783360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7158107060 con 0x7f71580f78e0 2023-11-30T21:45:24.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.205+0000 7f7154ff9700 1 -- 172.21.15.49:0/2929783360 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7148014420 con 0x7f71580f78e0 2023-11-30T21:45:24.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.205+0000 7f715d39f700 1 -- 172.21.15.49:0/2929783360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7158045ad0 con 0x7f71580f78e0 2023-11-30T21:45:24.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.205+0000 7f7154ff9700 1 --2- 172.21.15.49:0/2929783360 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f714002ff60 0x7f7140032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:24.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.205+0000 7f7154ff9700 1 -- 172.21.15.49:0/2929783360 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7148041030 con 0x7f71580f78e0 2023-11-30T21:45:24.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.205+0000 7f71577fe700 1 --2- 172.21.15.49:0/2929783360 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f714002ff60 0x7f7140032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:24.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.206+0000 7f71577fe700 1 --2- 172.21.15.49:0/2929783360 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f714002ff60 0x7f7140032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f714c0096a0 tx=0x7f714c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:24.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.210+0000 7f7154ff9700 1 -- 172.21.15.49:0/2929783360 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71480095f0 con 0x7f71580f78e0 2023-11-30T21:45:24.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.429+0000 7f715d39f700 1 -- 172.21.15.49:0/2929783360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7158107f10 con 0x7f71580f78e0 2023-11-30T21:45:24.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.430+0000 7f7154ff9700 1 -- 172.21.15.49:0/2929783360 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f714801cb10 con 0x7f71580f78e0 2023-11-30T21:45:24.431 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:24.431 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:24.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.431+0000 7f715d39f700 1 -- 172.21.15.49:0/2929783360 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f714002ff60 msgr2=0x7f7140032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:24.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.431+0000 7f715d39f700 1 --2- 172.21.15.49:0/2929783360 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f714002ff60 0x7f7140032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f714c0096a0 tx=0x7f714c006b40).stop 2023-11-30T21:45:24.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.431+0000 7f715d39f700 1 -- 172.21.15.49:0/2929783360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71580f78e0 msgr2=0x7f71581064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:24.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.431+0000 7f715d39f700 1 --2- 172.21.15.49:0/2929783360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71580f78e0 0x7f71581064e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f7148013510 tx=0x7f714800b040).stop 2023-11-30T21:45:24.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.431+0000 7f715d39f700 1 -- 172.21.15.49:0/2929783360 shutdown_connections 2023-11-30T21:45:24.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.431+0000 7f715d39f700 1 --2- 172.21.15.49:0/2929783360 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f714002ff60 0x7f7140032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:24.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.431+0000 7f715d39f700 1 --2- 172.21.15.49:0/2929783360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71580f78e0 0x7f71581064e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:24.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.431+0000 7f715d39f700 1 -- 172.21.15.49:0/2929783360 >> 172.21.15.49:0/2929783360 conn(0x7f71580f0120 msgr2=0x7f71580f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:24.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.431+0000 7f715d39f700 1 -- 172.21.15.49:0/2929783360 shutdown_connections 2023-11-30T21:45:24.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:24.431+0000 7f715d39f700 1 -- 172.21.15.49:0/2929783360 wait complete. 2023-11-30T21:45:24.434 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:25.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:23.467848+0000 mgr.smithi027.brrjcl (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:25.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:24.431235+0000 mon.smithi027 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.49:0/2929783360' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:25.818 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:25.819 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:27.087 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.085+0000 7f2fb7465700 1 -- 172.21.15.49:0/2723697017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb0060280 msgr2=0x7f2fb00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:27.088 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.085+0000 7f2fb7465700 1 --2- 172.21.15.49:0/2723697017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb0060280 0x7f2fb00f90f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2fac009fb0 tx=0x7f2fac009b30).stop 2023-11-30T21:45:27.088 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.085+0000 7f2fb7465700 1 -- 172.21.15.49:0/2723697017 shutdown_connections 2023-11-30T21:45:27.088 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.085+0000 7f2fb7465700 1 --2- 172.21.15.49:0/2723697017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb0060280 0x7f2fb00f90f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:27.088 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.085+0000 7f2fb7465700 1 -- 172.21.15.49:0/2723697017 >> 172.21.15.49:0/2723697017 conn(0x7f2fb00f0b70 msgr2=0x7f2fb00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:27.088 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.085+0000 7f2fb7465700 1 -- 172.21.15.49:0/2723697017 shutdown_connections 2023-11-30T21:45:27.088 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.086+0000 7f2fb7465700 1 -- 172.21.15.49:0/2723697017 wait complete. 2023-11-30T21:45:27.088 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.086+0000 7f2fb7465700 1 Processor -- start 2023-11-30T21:45:27.088 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.086+0000 7f2fb7465700 1 -- start start 2023-11-30T21:45:27.088 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.086+0000 7f2fb7465700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb0060280 0x7f2fb01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:27.088 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.086+0000 7f2fb7465700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb0106bd0 con 0x7f2fb0060280 2023-11-30T21:45:27.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.086+0000 7f2fb5201700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb0060280 0x7f2fb01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:27.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.086+0000 7f2fb5201700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb0060280 0x7f2fb01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:38000/0 (socket says 172.21.15.49:38000) 2023-11-30T21:45:27.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.086+0000 7f2fb5201700 1 -- 172.21.15.49:0/3250133580 learned_addr learned my addr 172.21.15.49:0/3250133580 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:27.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.087+0000 7f2fb5201700 1 -- 172.21.15.49:0/3250133580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac005040 con 0x7f2fb0060280 2023-11-30T21:45:27.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.087+0000 7f2fb5201700 1 --2- 172.21.15.49:0/3250133580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb0060280 0x7f2fb01066e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2fac013510 tx=0x7f2fac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:27.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.087+0000 7f2fa5ffb700 1 -- 172.21.15.49:0/3250133580 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fac009320 con 0x7f2fb0060280 2023-11-30T21:45:27.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.087+0000 7f2fa5ffb700 1 -- 172.21.15.49:0/3250133580 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fac013f00 con 0x7f2fb0060280 2023-11-30T21:45:27.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.087+0000 7f2fb7465700 1 -- 172.21.15.49:0/3250133580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb0106dd0 con 0x7f2fb0060280 2023-11-30T21:45:27.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.087+0000 7f2fa5ffb700 1 -- 172.21.15.49:0/3250133580 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fac014dc0 con 0x7f2fb0060280 2023-11-30T21:45:27.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.087+0000 7f2fb7465700 1 -- 172.21.15.49:0/3250133580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb0107260 con 0x7f2fb0060280 2023-11-30T21:45:27.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.088+0000 7f2fa5ffb700 1 -- 172.21.15.49:0/3250133580 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2fac01d3d0 con 0x7f2fb0060280 2023-11-30T21:45:27.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.088+0000 7f2fb7465700 1 -- 172.21.15.49:0/3250133580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb0045ad0 con 0x7f2fb0060280 2023-11-30T21:45:27.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.088+0000 7f2fa5ffb700 1 --2- 172.21.15.49:0/3250133580 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2f9c02ffb0 0x7f2f9c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:27.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.088+0000 7f2fa5ffb700 1 -- 172.21.15.49:0/3250133580 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2fac041110 con 0x7f2fb0060280 2023-11-30T21:45:27.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.089+0000 7f2fb4a00700 1 --2- 172.21.15.49:0/3250133580 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2f9c02ffb0 0x7f2f9c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:27.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.089+0000 7f2fb4a00700 1 --2- 172.21.15.49:0/3250133580 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2f9c02ffb0 0x7f2f9c032470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f2fa00096a0 tx=0x7f2fa0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:27.094 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.093+0000 7f2fa5ffb700 1 -- 172.21.15.49:0/3250133580 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2fac0126f0 con 0x7f2fb0060280 2023-11-30T21:45:27.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.311+0000 7f2fb7465700 1 -- 172.21.15.49:0/3250133580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fb0108110 con 0x7f2fb0060280 2023-11-30T21:45:27.313 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.311+0000 7f2fa5ffb700 1 -- 172.21.15.49:0/3250133580 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2fac012330 con 0x7f2fb0060280 2023-11-30T21:45:27.313 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:27.314 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:27.314 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.313+0000 7f2fb7465700 1 -- 172.21.15.49:0/3250133580 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2f9c02ffb0 msgr2=0x7f2f9c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:27.315 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.313+0000 7f2fb7465700 1 --2- 172.21.15.49:0/3250133580 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2f9c02ffb0 0x7f2f9c032470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f2fa00096a0 tx=0x7f2fa0006b40).stop 2023-11-30T21:45:27.315 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.313+0000 7f2fb7465700 1 -- 172.21.15.49:0/3250133580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb0060280 msgr2=0x7f2fb01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:27.315 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.313+0000 7f2fb7465700 1 --2- 172.21.15.49:0/3250133580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb0060280 0x7f2fb01066e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2fac013510 tx=0x7f2fac00b040).stop 2023-11-30T21:45:27.315 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.313+0000 7f2fb7465700 1 -- 172.21.15.49:0/3250133580 shutdown_connections 2023-11-30T21:45:27.315 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.313+0000 7f2fb7465700 1 --2- 172.21.15.49:0/3250133580 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2f9c02ffb0 0x7f2f9c032470 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:27.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.313+0000 7f2fb7465700 1 --2- 172.21.15.49:0/3250133580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2fb0060280 0x7f2fb01066e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:27.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.313+0000 7f2fb7465700 1 -- 172.21.15.49:0/3250133580 >> 172.21.15.49:0/3250133580 conn(0x7f2fb00f0b70 msgr2=0x7f2fb00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:27.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.313+0000 7f2fb7465700 1 -- 172.21.15.49:0/3250133580 shutdown_connections 2023-11-30T21:45:27.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:27.313+0000 7f2fb7465700 1 -- 172.21.15.49:0/3250133580 wait complete. 2023-11-30T21:45:27.317 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:27.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:27 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:25.468254+0000 mgr.smithi027.brrjcl (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:28.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:28 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:27.313057+0000 mon.smithi027 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.49:0/3250133580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:28.723 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:28.723 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:29.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:27.468561+0000 mgr.smithi027.brrjcl (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:29.955 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.953+0000 7fecf503d700 1 -- 172.21.15.49:0/4143499238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00f62a0 msgr2=0x7fecf00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:29.955 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.953+0000 7fecf503d700 1 --2- 172.21.15.49:0/4143499238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00f62a0 0x7fecf00f66c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fece0009fb0 tx=0x7fece0009b30).stop 2023-11-30T21:45:29.955 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.953+0000 7fecf503d700 1 -- 172.21.15.49:0/4143499238 shutdown_connections 2023-11-30T21:45:29.955 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.953+0000 7fecf503d700 1 --2- 172.21.15.49:0/4143499238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00f62a0 0x7fecf00f66c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:29.955 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.953+0000 7fecf503d700 1 -- 172.21.15.49:0/4143499238 >> 172.21.15.49:0/4143499238 conn(0x7fecf00f1aa0 msgr2=0x7fecf00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.953+0000 7fecf503d700 1 -- 172.21.15.49:0/4143499238 shutdown_connections 2023-11-30T21:45:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.953+0000 7fecf503d700 1 -- 172.21.15.49:0/4143499238 wait complete. 2023-11-30T21:45:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.953+0000 7fecf503d700 1 Processor -- start 2023-11-30T21:45:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.954+0000 7fecf503d700 1 -- start start 2023-11-30T21:45:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.954+0000 7fecf503d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00f62a0 0x7fecf01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.954+0000 7fecf503d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf01069a0 con 0x7fecf00f62a0 2023-11-30T21:45:29.957 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.954+0000 7feceffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00f62a0 0x7fecf01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:29.958 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.954+0000 7feceffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00f62a0 0x7fecf01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:60374/0 (socket says 172.21.15.49:60374) 2023-11-30T21:45:29.958 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.954+0000 7feceffff700 1 -- 172.21.15.49:0/403322011 learned_addr learned my addr 172.21.15.49:0/403322011 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:29.958 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.954+0000 7feceffff700 1 -- 172.21.15.49:0/403322011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece0005040 con 0x7fecf00f62a0 2023-11-30T21:45:29.958 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.955+0000 7feceffff700 1 --2- 172.21.15.49:0/403322011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00f62a0 0x7fecf01064b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fece0013510 tx=0x7fece000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:29.958 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.955+0000 7fececff9700 1 -- 172.21.15.49:0/403322011 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fece0009320 con 0x7fecf00f62a0 2023-11-30T21:45:29.958 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.955+0000 7fececff9700 1 -- 172.21.15.49:0/403322011 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fece0013f00 con 0x7fecf00f62a0 2023-11-30T21:45:29.958 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.955+0000 7fecf503d700 1 -- 172.21.15.49:0/403322011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf0106ba0 con 0x7fecf00f62a0 2023-11-30T21:45:29.958 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.955+0000 7fececff9700 1 -- 172.21.15.49:0/403322011 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fece0014dc0 con 0x7fecf00f62a0 2023-11-30T21:45:29.958 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.955+0000 7fecf503d700 1 -- 172.21.15.49:0/403322011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf0107030 con 0x7fecf00f62a0 2023-11-30T21:45:29.958 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.956+0000 7fececff9700 1 -- 172.21.15.49:0/403322011 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fece00095f0 con 0x7fecf00f62a0 2023-11-30T21:45:29.958 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.956+0000 7fecf503d700 1 -- 172.21.15.49:0/403322011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf0045ad0 con 0x7fecf00f62a0 2023-11-30T21:45:29.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.956+0000 7fececff9700 1 --2- 172.21.15.49:0/403322011 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fecd802ffb0 0x7fecd8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:29.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.956+0000 7fececff9700 1 -- 172.21.15.49:0/403322011 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fece0040ff0 con 0x7fecf00f62a0 2023-11-30T21:45:29.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.956+0000 7fecef7fe700 1 --2- 172.21.15.49:0/403322011 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fecd802ffb0 0x7fecd8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:29.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.956+0000 7fecef7fe700 1 --2- 172.21.15.49:0/403322011 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fecd802ffb0 0x7fecd8032470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fece40096a0 tx=0x7fece4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:29.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:29.961+0000 7fececff9700 1 -- 172.21.15.49:0/403322011 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fece006d2b0 con 0x7fecf00f62a0 2023-11-30T21:45:30.182 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:30.178+0000 7fecf503d700 1 -- 172.21.15.49:0/403322011 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fecf0107ee0 con 0x7fecf00f62a0 2023-11-30T21:45:30.183 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:30.179+0000 7fececff9700 1 -- 172.21.15.49:0/403322011 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fece0014420 con 0x7fecf00f62a0 2023-11-30T21:45:30.185 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:30.185 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:30.181+0000 7fecf503d700 1 -- 172.21.15.49:0/403322011 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fecd802ffb0 msgr2=0x7fecd8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:30.181+0000 7fecf503d700 1 --2- 172.21.15.49:0/403322011 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fecd802ffb0 0x7fecd8032470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fece40096a0 tx=0x7fece4006b40).stop 2023-11-30T21:45:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:30.181+0000 7fecf503d700 1 -- 172.21.15.49:0/403322011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00f62a0 msgr2=0x7fecf01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:30.181+0000 7fecf503d700 1 --2- 172.21.15.49:0/403322011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00f62a0 0x7fecf01064b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fece0013510 tx=0x7fece000b040).stop 2023-11-30T21:45:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:30.181+0000 7fecf503d700 1 -- 172.21.15.49:0/403322011 shutdown_connections 2023-11-30T21:45:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:30.181+0000 7fecf503d700 1 --2- 172.21.15.49:0/403322011 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fecd802ffb0 0x7fecd8032470 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:30.181+0000 7fecf503d700 1 --2- 172.21.15.49:0/403322011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecf00f62a0 0x7fecf01064b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:30.181+0000 7fecf503d700 1 -- 172.21.15.49:0/403322011 >> 172.21.15.49:0/403322011 conn(0x7fecf00f1aa0 msgr2=0x7fecf00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:30.181+0000 7fecf503d700 1 -- 172.21.15.49:0/403322011 shutdown_connections 2023-11-30T21:45:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:30.181+0000 7fecf503d700 1 -- 172.21.15.49:0/403322011 wait complete. 2023-11-30T21:45:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:31.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:29.468904+0000 mgr.smithi027.brrjcl (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:31.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:30.180632+0000 mon.smithi027 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.49:0/403322011' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:31.560 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:31.561 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:32.734 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.733+0000 7ff2e55bc700 1 -- 172.21.15.49:0/2287046906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e00f51a0 msgr2=0x7ff2e00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:32.734 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.733+0000 7ff2e55bc700 1 --2- 172.21.15.49:0/2287046906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e00f51a0 0x7ff2e00f55c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7ff2d0009fb0 tx=0x7ff2d0009b30).stop 2023-11-30T21:45:32.734 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.733+0000 7ff2e55bc700 1 -- 172.21.15.49:0/2287046906 shutdown_connections 2023-11-30T21:45:32.735 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.733+0000 7ff2e55bc700 1 --2- 172.21.15.49:0/2287046906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e00f51a0 0x7ff2e00f55c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:32.735 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.733+0000 7ff2e55bc700 1 -- 172.21.15.49:0/2287046906 >> 172.21.15.49:0/2287046906 conn(0x7ff2e00f09a0 msgr2=0x7ff2e00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:32.735 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.733+0000 7ff2e55bc700 1 -- 172.21.15.49:0/2287046906 shutdown_connections 2023-11-30T21:45:32.735 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.733+0000 7ff2e55bc700 1 -- 172.21.15.49:0/2287046906 wait complete. 2023-11-30T21:45:32.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.733+0000 7ff2e55bc700 1 Processor -- start 2023-11-30T21:45:32.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.733+0000 7ff2e55bc700 1 -- start start 2023-11-30T21:45:32.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.734+0000 7ff2e55bc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e00f51a0 0x7ff2e01091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:32.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.734+0000 7ff2e55bc700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e0060240 con 0x7ff2e00f51a0 2023-11-30T21:45:32.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.734+0000 7ff2dffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e00f51a0 0x7ff2e01091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:32.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.734+0000 7ff2dffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e00f51a0 0x7ff2e01091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:60388/0 (socket says 172.21.15.49:60388) 2023-11-30T21:45:32.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.734+0000 7ff2dffff700 1 -- 172.21.15.49:0/3709199581 learned_addr learned my addr 172.21.15.49:0/3709199581 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:32.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.734+0000 7ff2dffff700 1 -- 172.21.15.49:0/3709199581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d0005040 con 0x7ff2e00f51a0 2023-11-30T21:45:32.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.734+0000 7ff2dffff700 1 --2- 172.21.15.49:0/3709199581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e00f51a0 0x7ff2e01091b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff2d0013510 tx=0x7ff2d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:32.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.735+0000 7ff2dcff9700 1 -- 172.21.15.49:0/3709199581 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2d0009320 con 0x7ff2e00f51a0 2023-11-30T21:45:32.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.735+0000 7ff2e55bc700 1 -- 172.21.15.49:0/3709199581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e0060440 con 0x7ff2e00f51a0 2023-11-30T21:45:32.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.735+0000 7ff2dcff9700 1 -- 172.21.15.49:0/3709199581 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2d0013f00 con 0x7ff2e00f51a0 2023-11-30T21:45:32.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.735+0000 7ff2dcff9700 1 -- 172.21.15.49:0/3709199581 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2d0014dc0 con 0x7ff2e00f51a0 2023-11-30T21:45:32.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.735+0000 7ff2e55bc700 1 -- 172.21.15.49:0/3709199581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e0109860 con 0x7ff2e00f51a0 2023-11-30T21:45:32.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.735+0000 7ff2dcff9700 1 -- 172.21.15.49:0/3709199581 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2d0014420 con 0x7ff2e00f51a0 2023-11-30T21:45:32.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.735+0000 7ff2e55bc700 1 -- 172.21.15.49:0/3709199581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e0045ad0 con 0x7ff2e00f51a0 2023-11-30T21:45:32.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.736+0000 7ff2dcff9700 1 --2- 172.21.15.49:0/3709199581 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff2c802ff60 0x7ff2c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:32.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.736+0000 7ff2dcff9700 1 -- 172.21.15.49:0/3709199581 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2d0041110 con 0x7ff2e00f51a0 2023-11-30T21:45:32.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.736+0000 7ff2df7fe700 1 --2- 172.21.15.49:0/3709199581 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff2c802ff60 0x7ff2c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:32.738 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.736+0000 7ff2df7fe700 1 --2- 172.21.15.49:0/3709199581 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff2c802ff60 0x7ff2c8032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff2d40096a0 tx=0x7ff2d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:32.742 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.740+0000 7ff2dcff9700 1 -- 172.21.15.49:0/3709199581 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2d00095f0 con 0x7ff2e00f51a0 2023-11-30T21:45:32.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.961+0000 7ff2e55bc700 1 -- 172.21.15.49:0/3709199581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2e010a080 con 0x7ff2e00f51a0 2023-11-30T21:45:32.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.962+0000 7ff2dcff9700 1 -- 172.21.15.49:0/3709199581 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff2d001cb10 con 0x7ff2e00f51a0 2023-11-30T21:45:32.963 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:32.963 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:32.964 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.963+0000 7ff2e55bc700 1 -- 172.21.15.49:0/3709199581 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff2c802ff60 msgr2=0x7ff2c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:32.965 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.963+0000 7ff2e55bc700 1 --2- 172.21.15.49:0/3709199581 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff2c802ff60 0x7ff2c8032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff2d40096a0 tx=0x7ff2d4006b40).stop 2023-11-30T21:45:32.965 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.963+0000 7ff2e55bc700 1 -- 172.21.15.49:0/3709199581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e00f51a0 msgr2=0x7ff2e01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:32.965 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.963+0000 7ff2e55bc700 1 --2- 172.21.15.49:0/3709199581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e00f51a0 0x7ff2e01091b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7ff2d0013510 tx=0x7ff2d000b040).stop 2023-11-30T21:45:32.965 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.963+0000 7ff2e55bc700 1 -- 172.21.15.49:0/3709199581 shutdown_connections 2023-11-30T21:45:32.965 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.963+0000 7ff2e55bc700 1 --2- 172.21.15.49:0/3709199581 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff2c802ff60 0x7ff2c8032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:32.965 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.963+0000 7ff2e55bc700 1 --2- 172.21.15.49:0/3709199581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2e00f51a0 0x7ff2e01091b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:32.965 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.963+0000 7ff2e55bc700 1 -- 172.21.15.49:0/3709199581 >> 172.21.15.49:0/3709199581 conn(0x7ff2e00f09a0 msgr2=0x7ff2e00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:32.965 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.963+0000 7ff2e55bc700 1 -- 172.21.15.49:0/3709199581 shutdown_connections 2023-11-30T21:45:32.965 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:32.963+0000 7ff2e55bc700 1 -- 172.21.15.49:0/3709199581 wait complete. 2023-11-30T21:45:32.966 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:33.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:33 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:31.469298+0000 mgr.smithi027.brrjcl (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:33.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:33 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:32.963009+0000 mon.smithi027 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.49:0/3709199581' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:34.578 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:34.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:35.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:35 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:33.469589+0000 mgr.smithi027.brrjcl (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:35.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.720+0000 7fb55346c700 1 -- 172.21.15.49:0/2298946162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c0f51a0 msgr2=0x7fb54c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:35.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.720+0000 7fb55346c700 1 --2- 172.21.15.49:0/2298946162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c0f51a0 0x7fb54c0f55c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fb548009fb0 tx=0x7fb548009b30).stop 2023-11-30T21:45:35.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.720+0000 7fb55346c700 1 -- 172.21.15.49:0/2298946162 shutdown_connections 2023-11-30T21:45:35.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.720+0000 7fb55346c700 1 --2- 172.21.15.49:0/2298946162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c0f51a0 0x7fb54c0f55c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:35.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.720+0000 7fb55346c700 1 -- 172.21.15.49:0/2298946162 >> 172.21.15.49:0/2298946162 conn(0x7fb54c0f09a0 msgr2=0x7fb54c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:35.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.720+0000 7fb55346c700 1 -- 172.21.15.49:0/2298946162 shutdown_connections 2023-11-30T21:45:35.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.720+0000 7fb55346c700 1 -- 172.21.15.49:0/2298946162 wait complete. 2023-11-30T21:45:35.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.720+0000 7fb55346c700 1 Processor -- start 2023-11-30T21:45:35.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.720+0000 7fb55346c700 1 -- start start 2023-11-30T21:45:35.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.721+0000 7fb55346c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c0f51a0 0x7fb54c0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:35.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.721+0000 7fb55346c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c0fb5a0 con 0x7fb54c0f51a0 2023-11-30T21:45:35.723 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.721+0000 7fb55246a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c0f51a0 0x7fb54c0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:35.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.721+0000 7fb55246a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c0f51a0 0x7fb54c0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:60398/0 (socket says 172.21.15.49:60398) 2023-11-30T21:45:35.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.721+0000 7fb55246a700 1 -- 172.21.15.49:0/242590969 learned_addr learned my addr 172.21.15.49:0/242590969 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:35.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.721+0000 7fb55246a700 1 -- 172.21.15.49:0/242590969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb548005040 con 0x7fb54c0f51a0 2023-11-30T21:45:35.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.721+0000 7fb55246a700 1 --2- 172.21.15.49:0/242590969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c0f51a0 0x7fb54c0fd120 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fb548013510 tx=0x7fb54800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:35.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.721+0000 7fb53effd700 1 -- 172.21.15.49:0/242590969 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb548009320 con 0x7fb54c0f51a0 2023-11-30T21:45:35.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.721+0000 7fb55346c700 1 -- 172.21.15.49:0/242590969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54c0fb740 con 0x7fb54c0f51a0 2023-11-30T21:45:35.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.721+0000 7fb53effd700 1 -- 172.21.15.49:0/242590969 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb548013f00 con 0x7fb54c0f51a0 2023-11-30T21:45:35.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.721+0000 7fb53effd700 1 -- 172.21.15.49:0/242590969 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb548014dc0 con 0x7fb54c0f51a0 2023-11-30T21:45:35.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.722+0000 7fb55346c700 1 -- 172.21.15.49:0/242590969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54c0fbbd0 con 0x7fb54c0f51a0 2023-11-30T21:45:35.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.722+0000 7fb53effd700 1 -- 172.21.15.49:0/242590969 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5480095f0 con 0x7fb54c0f51a0 2023-11-30T21:45:35.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.722+0000 7fb55346c700 1 -- 172.21.15.49:0/242590969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54c059100 con 0x7fb54c0f51a0 2023-11-30T21:45:35.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.722+0000 7fb53effd700 1 --2- 172.21.15.49:0/242590969 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb53802ff60 0x7fb538032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:35.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.723+0000 7fb53effd700 1 -- 172.21.15.49:0/242590969 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb5480410c0 con 0x7fb54c0f51a0 2023-11-30T21:45:35.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.723+0000 7fb551c69700 1 --2- 172.21.15.49:0/242590969 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb53802ff60 0x7fb538032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:35.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.723+0000 7fb551c69700 1 --2- 172.21.15.49:0/242590969 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb53802ff60 0x7fb538032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fb5400096a0 tx=0x7fb540006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:35.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.728+0000 7fb53effd700 1 -- 172.21.15.49:0/242590969 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb548014420 con 0x7fb54c0f51a0 2023-11-30T21:45:35.950 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.948+0000 7fb55346c700 1 -- 172.21.15.49:0/242590969 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb54c0fabe0 con 0x7fb54c0f51a0 2023-11-30T21:45:35.950 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.949+0000 7fb53effd700 1 -- 172.21.15.49:0/242590969 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb548010230 con 0x7fb54c0f51a0 2023-11-30T21:45:35.950 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:35.950 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:35.952 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.950+0000 7fb55346c700 1 -- 172.21.15.49:0/242590969 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb53802ff60 msgr2=0x7fb538032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:35.952 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.950+0000 7fb55346c700 1 --2- 172.21.15.49:0/242590969 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb53802ff60 0x7fb538032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fb5400096a0 tx=0x7fb540006b40).stop 2023-11-30T21:45:35.952 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.950+0000 7fb55346c700 1 -- 172.21.15.49:0/242590969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c0f51a0 msgr2=0x7fb54c0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:35.952 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.950+0000 7fb55346c700 1 --2- 172.21.15.49:0/242590969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c0f51a0 0x7fb54c0fd120 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fb548013510 tx=0x7fb54800b040).stop 2023-11-30T21:45:35.952 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.950+0000 7fb55346c700 1 -- 172.21.15.49:0/242590969 shutdown_connections 2023-11-30T21:45:35.952 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.950+0000 7fb55346c700 1 --2- 172.21.15.49:0/242590969 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb53802ff60 0x7fb538032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:35.952 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.950+0000 7fb55346c700 1 --2- 172.21.15.49:0/242590969 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54c0f51a0 0x7fb54c0fd120 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:35.952 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.950+0000 7fb55346c700 1 -- 172.21.15.49:0/242590969 >> 172.21.15.49:0/242590969 conn(0x7fb54c0f09a0 msgr2=0x7fb54c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:35.953 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.950+0000 7fb55346c700 1 -- 172.21.15.49:0/242590969 shutdown_connections 2023-11-30T21:45:35.953 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:35.950+0000 7fb55346c700 1 -- 172.21.15.49:0/242590969 wait complete. 2023-11-30T21:45:35.953 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:36.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:35.950146+0000 mon.smithi027 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.49:0/242590969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:37.340 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:37.341 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:37.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:37 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:35.469865+0000 mgr.smithi027.brrjcl (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:38.811 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.732+0000 7fcb64b15700 1 -- 172.21.15.49:0/4046770927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb600f5410 msgr2=0x7fcb600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:38.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.732+0000 7fcb64b15700 1 --2- 172.21.15.49:0/4046770927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb600f5410 0x7fcb600f5830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fcb50009fb0 tx=0x7fcb50009b30).stop 2023-11-30T21:45:38.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.732+0000 7fcb64b15700 1 -- 172.21.15.49:0/4046770927 shutdown_connections 2023-11-30T21:45:38.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.732+0000 7fcb64b15700 1 --2- 172.21.15.49:0/4046770927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb600f5410 0x7fcb600f5830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:38.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.732+0000 7fcb64b15700 1 -- 172.21.15.49:0/4046770927 >> 172.21.15.49:0/4046770927 conn(0x7fcb600f0b70 msgr2=0x7fcb600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:38.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.732+0000 7fcb64b15700 1 -- 172.21.15.49:0/4046770927 shutdown_connections 2023-11-30T21:45:38.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.732+0000 7fcb64b15700 1 -- 172.21.15.49:0/4046770927 wait complete. 2023-11-30T21:45:38.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.733+0000 7fcb64b15700 1 Processor -- start 2023-11-30T21:45:38.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.733+0000 7fcb64b15700 1 -- start start 2023-11-30T21:45:38.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.733+0000 7fcb64b15700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb600f5410 0x7fcb600fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:38.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.733+0000 7fcb64b15700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb600fb820 con 0x7fcb600f5410 2023-11-30T21:45:38.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.733+0000 7fcb5e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb600f5410 0x7fcb600fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:38.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.733+0000 7fcb5e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb600f5410 0x7fcb600fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:47838/0 (socket says 172.21.15.49:47838) 2023-11-30T21:45:38.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.733+0000 7fcb5e59c700 1 -- 172.21.15.49:0/1742142801 learned_addr learned my addr 172.21.15.49:0/1742142801 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:38.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.734+0000 7fcb5e59c700 1 -- 172.21.15.49:0/1742142801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb50005040 con 0x7fcb600f5410 2023-11-30T21:45:38.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.734+0000 7fcb5e59c700 1 --2- 172.21.15.49:0/1742142801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb600f5410 0x7fcb600fd3a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fcb50013510 tx=0x7fcb5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:38.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.734+0000 7fcb56ffd700 1 -- 172.21.15.49:0/1742142801 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb50009320 con 0x7fcb600f5410 2023-11-30T21:45:38.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.734+0000 7fcb64b15700 1 -- 172.21.15.49:0/1742142801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb600fb9c0 con 0x7fcb600f5410 2023-11-30T21:45:38.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.734+0000 7fcb56ffd700 1 -- 172.21.15.49:0/1742142801 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb50013f00 con 0x7fcb600f5410 2023-11-30T21:45:38.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.734+0000 7fcb56ffd700 1 -- 172.21.15.49:0/1742142801 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb50014d90 con 0x7fcb600f5410 2023-11-30T21:45:38.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.734+0000 7fcb64b15700 1 -- 172.21.15.49:0/1742142801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb600fbe50 con 0x7fcb600f5410 2023-11-30T21:45:38.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.735+0000 7fcb56ffd700 1 -- 172.21.15.49:0/1742142801 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb500263a0 con 0x7fcb600f5410 2023-11-30T21:45:38.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.735+0000 7fcb56ffd700 1 --2- 172.21.15.49:0/1742142801 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcb4c02ff60 0x7fcb4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:38.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.735+0000 7fcb56ffd700 1 -- 172.21.15.49:0/1742142801 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb50040fd0 con 0x7fcb600f5410 2023-11-30T21:45:38.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.735+0000 7fcb64b15700 1 -- 172.21.15.49:0/1742142801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb60059100 con 0x7fcb600f5410 2023-11-30T21:45:38.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.735+0000 7fcb5dd9b700 1 --2- 172.21.15.49:0/1742142801 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcb4c02ff60 0x7fcb4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:38.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.736+0000 7fcb5dd9b700 1 --2- 172.21.15.49:0/1742142801 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcb4c02ff60 0x7fcb4c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fcb480096a0 tx=0x7fcb48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:38.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.740+0000 7fcb56ffd700 1 -- 172.21.15.49:0/1742142801 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb50012920 con 0x7fcb600f5410 2023-11-30T21:45:38.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.957+0000 7fcb64b15700 1 -- 172.21.15.49:0/1742142801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb60045ad0 con 0x7fcb600f5410 2023-11-30T21:45:38.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.958+0000 7fcb56ffd700 1 -- 172.21.15.49:0/1742142801 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcb5002b020 con 0x7fcb600f5410 2023-11-30T21:45:38.960 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:38.960 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:38.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.960+0000 7fcb64b15700 1 -- 172.21.15.49:0/1742142801 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcb4c02ff60 msgr2=0x7fcb4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:38.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.960+0000 7fcb64b15700 1 --2- 172.21.15.49:0/1742142801 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcb4c02ff60 0x7fcb4c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fcb480096a0 tx=0x7fcb48006b40).stop 2023-11-30T21:45:38.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.960+0000 7fcb64b15700 1 -- 172.21.15.49:0/1742142801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb600f5410 msgr2=0x7fcb600fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:38.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.960+0000 7fcb64b15700 1 --2- 172.21.15.49:0/1742142801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb600f5410 0x7fcb600fd3a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fcb50013510 tx=0x7fcb5000b040).stop 2023-11-30T21:45:38.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.960+0000 7fcb64b15700 1 -- 172.21.15.49:0/1742142801 shutdown_connections 2023-11-30T21:45:38.962 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.960+0000 7fcb64b15700 1 --2- 172.21.15.49:0/1742142801 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcb4c02ff60 0x7fcb4c032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:38.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.960+0000 7fcb64b15700 1 --2- 172.21.15.49:0/1742142801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb600f5410 0x7fcb600fd3a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:38.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.960+0000 7fcb64b15700 1 -- 172.21.15.49:0/1742142801 >> 172.21.15.49:0/1742142801 conn(0x7fcb600f0b70 msgr2=0x7fcb6005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:38.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.960+0000 7fcb64b15700 1 -- 172.21.15.49:0/1742142801 shutdown_connections 2023-11-30T21:45:38.963 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:38.960+0000 7fcb64b15700 1 -- 172.21.15.49:0/1742142801 wait complete. 2023-11-30T21:45:38.963 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:39.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:37.470142+0000 mgr.smithi027.brrjcl (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:39.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:38.959472+0000 mon.smithi027 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.49:0/1742142801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:41.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:41 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:39.470468+0000 mgr.smithi027.brrjcl (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:41.750 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:41.751 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:43.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:41.470755+0000 mgr.smithi027.brrjcl (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:45.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.491+0000 7fe630174700 1 -- 172.21.15.49:0/3321876950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628060200 msgr2=0x7fe6280f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:45.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.491+0000 7fe630174700 1 --2- 172.21.15.49:0/3321876950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628060200 0x7fe6280f9170 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fe624009fb0 tx=0x7fe624009b30).stop 2023-11-30T21:45:45.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.491+0000 7fe630174700 1 -- 172.21.15.49:0/3321876950 shutdown_connections 2023-11-30T21:45:45.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.491+0000 7fe630174700 1 --2- 172.21.15.49:0/3321876950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628060200 0x7fe6280f9170 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:45.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.491+0000 7fe630174700 1 -- 172.21.15.49:0/3321876950 >> 172.21.15.49:0/3321876950 conn(0x7fe6280f0b70 msgr2=0x7fe6280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:45.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.491+0000 7fe630174700 1 -- 172.21.15.49:0/3321876950 shutdown_connections 2023-11-30T21:45:45.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.491+0000 7fe630174700 1 -- 172.21.15.49:0/3321876950 wait complete. 2023-11-30T21:45:45.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.492+0000 7fe630174700 1 Processor -- start 2023-11-30T21:45:45.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.492+0000 7fe630174700 1 -- start start 2023-11-30T21:45:45.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.492+0000 7fe630174700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628060200 0x7fe6281066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:45.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.492+0000 7fe630174700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe628106b90 con 0x7fe628060200 2023-11-30T21:45:45.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.492+0000 7fe62df10700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628060200 0x7fe6281066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:45.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.492+0000 7fe62df10700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628060200 0x7fe6281066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:47858/0 (socket says 172.21.15.49:47858) 2023-11-30T21:45:45.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.492+0000 7fe62df10700 1 -- 172.21.15.49:0/2259500057 learned_addr learned my addr 172.21.15.49:0/2259500057 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:45.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.493+0000 7fe62df10700 1 -- 172.21.15.49:0/2259500057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe624005040 con 0x7fe628060200 2023-11-30T21:45:45.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.493+0000 7fe62df10700 1 --2- 172.21.15.49:0/2259500057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628060200 0x7fe6281066a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe624009830 tx=0x7fe62400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:45.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.493+0000 7fe61e7fc700 1 -- 172.21.15.49:0/2259500057 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe624009320 con 0x7fe628060200 2023-11-30T21:45:45.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.493+0000 7fe630174700 1 -- 172.21.15.49:0/2259500057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe628106d90 con 0x7fe628060200 2023-11-30T21:45:45.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.493+0000 7fe61e7fc700 1 -- 172.21.15.49:0/2259500057 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe6240136c0 con 0x7fe628060200 2023-11-30T21:45:45.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.493+0000 7fe61e7fc700 1 -- 172.21.15.49:0/2259500057 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe62401ee90 con 0x7fe628060200 2023-11-30T21:45:45.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.493+0000 7fe630174700 1 -- 172.21.15.49:0/2259500057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe628107220 con 0x7fe628060200 2023-11-30T21:45:45.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.494+0000 7fe61e7fc700 1 -- 172.21.15.49:0/2259500057 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe62401e420 con 0x7fe628060200 2023-11-30T21:45:45.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.494+0000 7fe630174700 1 -- 172.21.15.49:0/2259500057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe628045ad0 con 0x7fe628060200 2023-11-30T21:45:45.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.494+0000 7fe61e7fc700 1 --2- 172.21.15.49:0/2259500057 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe61402ff10 0x7fe6140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:45.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.494+0000 7fe61e7fc700 1 -- 172.21.15.49:0/2259500057 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe624040f40 con 0x7fe628060200 2023-11-30T21:45:45.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.494+0000 7fe62d70f700 1 --2- 172.21.15.49:0/2259500057 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe61402ff10 0x7fe6140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:45.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.495+0000 7fe62d70f700 1 --2- 172.21.15.49:0/2259500057 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe61402ff10 0x7fe6140323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe6180096a0 tx=0x7fe618006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:45.499 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.499+0000 7fe61e7fc700 1 -- 172.21.15.49:0/2259500057 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe62406d2b0 con 0x7fe628060200 2023-11-30T21:45:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:45 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:43.471011+0000 mgr.smithi027.brrjcl (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:45.719 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.716+0000 7fe630174700 1 -- 172.21.15.49:0/2259500057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6281080d0 con 0x7fe628060200 2023-11-30T21:45:45.719 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.716+0000 7fe61e7fc700 1 -- 172.21.15.49:0/2259500057 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe6240095f0 con 0x7fe628060200 2023-11-30T21:45:45.720 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:45.720 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:45.720 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.718+0000 7fe630174700 1 -- 172.21.15.49:0/2259500057 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe61402ff10 msgr2=0x7fe6140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:45.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.718+0000 7fe630174700 1 --2- 172.21.15.49:0/2259500057 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe61402ff10 0x7fe6140323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe6180096a0 tx=0x7fe618006b40).stop 2023-11-30T21:45:45.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.718+0000 7fe630174700 1 -- 172.21.15.49:0/2259500057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628060200 msgr2=0x7fe6281066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:45.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.718+0000 7fe630174700 1 --2- 172.21.15.49:0/2259500057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628060200 0x7fe6281066a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe624009830 tx=0x7fe62400b040).stop 2023-11-30T21:45:45.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.718+0000 7fe630174700 1 -- 172.21.15.49:0/2259500057 shutdown_connections 2023-11-30T21:45:45.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.718+0000 7fe630174700 1 --2- 172.21.15.49:0/2259500057 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe61402ff10 0x7fe6140323d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:45.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.718+0000 7fe630174700 1 --2- 172.21.15.49:0/2259500057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe628060200 0x7fe6281066a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:45.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.718+0000 7fe630174700 1 -- 172.21.15.49:0/2259500057 >> 172.21.15.49:0/2259500057 conn(0x7fe6280f0b70 msgr2=0x7fe6280f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:45.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.718+0000 7fe630174700 1 -- 172.21.15.49:0/2259500057 shutdown_connections 2023-11-30T21:45:45.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:45.718+0000 7fe630174700 1 -- 172.21.15.49:0/2259500057 wait complete. 2023-11-30T21:45:45.729 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:46.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:46 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:45.716713+0000 mon.smithi027 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.49:0/2259500057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:47.130 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:47.131 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:47.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:47 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:45.471356+0000 mgr.smithi027.brrjcl (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:48.358 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.357+0000 7f820d1fb700 1 -- 172.21.15.49:0/4152521812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82080f62c0 msgr2=0x7f82080f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:48.358 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.357+0000 7f820d1fb700 1 --2- 172.21.15.49:0/4152521812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82080f62c0 0x7f82080f66e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f81f8009fb0 tx=0x7f81f8009b30).stop 2023-11-30T21:45:48.358 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.357+0000 7f820d1fb700 1 -- 172.21.15.49:0/4152521812 shutdown_connections 2023-11-30T21:45:48.358 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.357+0000 7f820d1fb700 1 --2- 172.21.15.49:0/4152521812 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82080f62c0 0x7f82080f66e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:48.359 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.357+0000 7f820d1fb700 1 -- 172.21.15.49:0/4152521812 >> 172.21.15.49:0/4152521812 conn(0x7f82080f1ac0 msgr2=0x7f82080f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:48.359 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.357+0000 7f820d1fb700 1 -- 172.21.15.49:0/4152521812 shutdown_connections 2023-11-30T21:45:48.359 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.357+0000 7f820d1fb700 1 -- 172.21.15.49:0/4152521812 wait complete. 2023-11-30T21:45:48.362 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.358+0000 7f820d1fb700 1 Processor -- start 2023-11-30T21:45:48.362 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.358+0000 7f820d1fb700 1 -- start start 2023-11-30T21:45:48.362 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.358+0000 7f820d1fb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82080f62c0 0x7f8208106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:48.362 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.358+0000 7f820d1fb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8208106a10 con 0x7f82080f62c0 2023-11-30T21:45:48.362 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.358+0000 7f8207fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82080f62c0 0x7f8208106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:48.362 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.358+0000 7f8207fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82080f62c0 0x7f8208106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:47870/0 (socket says 172.21.15.49:47870) 2023-11-30T21:45:48.362 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.358+0000 7f8207fff700 1 -- 172.21.15.49:0/1408195024 learned_addr learned my addr 172.21.15.49:0/1408195024 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:48.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.359+0000 7f8207fff700 1 -- 172.21.15.49:0/1408195024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f8005040 con 0x7f82080f62c0 2023-11-30T21:45:48.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.359+0000 7f8207fff700 1 --2- 172.21.15.49:0/1408195024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82080f62c0 0x7f8208106520 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f81f8013510 tx=0x7f81f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:48.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.359+0000 7f8204ff9700 1 -- 172.21.15.49:0/1408195024 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81f8009320 con 0x7f82080f62c0 2023-11-30T21:45:48.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.359+0000 7f820d1fb700 1 -- 172.21.15.49:0/1408195024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8208106c10 con 0x7f82080f62c0 2023-11-30T21:45:48.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.359+0000 7f8204ff9700 1 -- 172.21.15.49:0/1408195024 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81f8013f00 con 0x7f82080f62c0 2023-11-30T21:45:48.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.359+0000 7f8204ff9700 1 -- 172.21.15.49:0/1408195024 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81f8014dc0 con 0x7f82080f62c0 2023-11-30T21:45:48.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.359+0000 7f820d1fb700 1 -- 172.21.15.49:0/1408195024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82081070a0 con 0x7f82080f62c0 2023-11-30T21:45:48.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.360+0000 7f8204ff9700 1 -- 172.21.15.49:0/1408195024 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81f801d3d0 con 0x7f82080f62c0 2023-11-30T21:45:48.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.360+0000 7f8204ff9700 1 --2- 172.21.15.49:0/1408195024 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f81f002ff60 0x7f81f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:48.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.360+0000 7f8204ff9700 1 -- 172.21.15.49:0/1408195024 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f81f8041050 con 0x7f82080f62c0 2023-11-30T21:45:48.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.360+0000 7f820d1fb700 1 -- 172.21.15.49:0/1408195024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8208059100 con 0x7f82080f62c0 2023-11-30T21:45:48.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.360+0000 7f82077fe700 1 --2- 172.21.15.49:0/1408195024 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f81f002ff60 0x7f81f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:48.364 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.361+0000 7f82077fe700 1 --2- 172.21.15.49:0/1408195024 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f81f002ff60 0x7f81f0032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f81fc0096a0 tx=0x7f81fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:48.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.365+0000 7f8204ff9700 1 -- 172.21.15.49:0/1408195024 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81f8012920 con 0x7f82080f62c0 2023-11-30T21:45:48.582 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.582+0000 7f820d1fb700 1 -- 172.21.15.49:0/1408195024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f82080fbd00 con 0x7f82080f62c0 2023-11-30T21:45:48.583 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.582+0000 7f8204ff9700 1 -- 172.21.15.49:0/1408195024 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f81f802b020 con 0x7f82080f62c0 2023-11-30T21:45:48.583 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:48.583 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:48.584 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.584+0000 7f820d1fb700 1 -- 172.21.15.49:0/1408195024 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f81f002ff60 msgr2=0x7f81f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:48.584 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.584+0000 7f820d1fb700 1 --2- 172.21.15.49:0/1408195024 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f81f002ff60 0x7f81f0032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f81fc0096a0 tx=0x7f81fc006b40).stop 2023-11-30T21:45:48.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.584+0000 7f820d1fb700 1 -- 172.21.15.49:0/1408195024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82080f62c0 msgr2=0x7f8208106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:48.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.584+0000 7f820d1fb700 1 --2- 172.21.15.49:0/1408195024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82080f62c0 0x7f8208106520 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f81f8013510 tx=0x7f81f800b040).stop 2023-11-30T21:45:48.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.584+0000 7f820d1fb700 1 -- 172.21.15.49:0/1408195024 shutdown_connections 2023-11-30T21:45:48.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.584+0000 7f820d1fb700 1 --2- 172.21.15.49:0/1408195024 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f81f002ff60 0x7f81f0032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:48.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.584+0000 7f820d1fb700 1 --2- 172.21.15.49:0/1408195024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82080f62c0 0x7f8208106520 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:48.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.584+0000 7f820d1fb700 1 -- 172.21.15.49:0/1408195024 >> 172.21.15.49:0/1408195024 conn(0x7f82080f1ac0 msgr2=0x7f82080f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:48.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.584+0000 7f820d1fb700 1 -- 172.21.15.49:0/1408195024 shutdown_connections 2023-11-30T21:45:48.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:48.584+0000 7f820d1fb700 1 -- 172.21.15.49:0/1408195024 wait complete. 2023-11-30T21:45:48.586 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:49.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:49 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:47.471691+0000 mgr.smithi027.brrjcl (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:49.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:49 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:48.583009+0000 mon.smithi027 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.49:0/1408195024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:49.992 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:49.992 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:51.238 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.236+0000 7fcff82b3700 1 -- 172.21.15.49:0/2666486738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff00f6950 msgr2=0x7fcff00f6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:51.238 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.236+0000 7fcff82b3700 1 --2- 172.21.15.49:0/2666486738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff00f6950 0x7fcff00f6d70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fcfec009fb0 tx=0x7fcfec009b30).stop 2023-11-30T21:45:51.238 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.236+0000 7fcff82b3700 1 -- 172.21.15.49:0/2666486738 shutdown_connections 2023-11-30T21:45:51.238 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.236+0000 7fcff82b3700 1 --2- 172.21.15.49:0/2666486738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff00f6950 0x7fcff00f6d70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:51.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.236+0000 7fcff82b3700 1 -- 172.21.15.49:0/2666486738 >> 172.21.15.49:0/2666486738 conn(0x7fcff00f0260 msgr2=0x7fcff00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:51.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.236+0000 7fcff82b3700 1 -- 172.21.15.49:0/2666486738 shutdown_connections 2023-11-30T21:45:51.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.237+0000 7fcff82b3700 1 -- 172.21.15.49:0/2666486738 wait complete. 2023-11-30T21:45:51.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.237+0000 7fcff82b3700 1 Processor -- start 2023-11-30T21:45:51.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.237+0000 7fcff82b3700 1 -- start start 2023-11-30T21:45:51.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.237+0000 7fcff82b3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff00f6950 0x7fcff01055e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:51.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.237+0000 7fcff82b3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff0105ad0 con 0x7fcff00f6950 2023-11-30T21:45:51.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.237+0000 7fcff604f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff00f6950 0x7fcff01055e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:51.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.238+0000 7fcff604f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff00f6950 0x7fcff01055e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:33148/0 (socket says 172.21.15.49:33148) 2023-11-30T21:45:51.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.238+0000 7fcff604f700 1 -- 172.21.15.49:0/1934699607 learned_addr learned my addr 172.21.15.49:0/1934699607 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:51.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.238+0000 7fcff604f700 1 -- 172.21.15.49:0/1934699607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfec005040 con 0x7fcff00f6950 2023-11-30T21:45:51.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.238+0000 7fcff604f700 1 --2- 172.21.15.49:0/1934699607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff00f6950 0x7fcff01055e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fcfec014000 tx=0x7fcfec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:51.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.238+0000 7fcfe6ffd700 1 -- 172.21.15.49:0/1934699607 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfec0095f0 con 0x7fcff00f6950 2023-11-30T21:45:51.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.238+0000 7fcff82b3700 1 -- 172.21.15.49:0/1934699607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff0105cd0 con 0x7fcff00f6950 2023-11-30T21:45:51.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.238+0000 7fcfe6ffd700 1 -- 172.21.15.49:0/1934699607 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfec007e70 con 0x7fcff00f6950 2023-11-30T21:45:51.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.238+0000 7fcfe6ffd700 1 -- 172.21.15.49:0/1934699607 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfec014ea0 con 0x7fcff00f6950 2023-11-30T21:45:51.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.238+0000 7fcff82b3700 1 -- 172.21.15.49:0/1934699607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff0106160 con 0x7fcff00f6950 2023-11-30T21:45:51.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.239+0000 7fcff82b3700 1 -- 172.21.15.49:0/1934699607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff0045ad0 con 0x7fcff00f6950 2023-11-30T21:45:51.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.239+0000 7fcfe6ffd700 1 -- 172.21.15.49:0/1934699607 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfec025070 con 0x7fcff00f6950 2023-11-30T21:45:51.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.239+0000 7fcfe6ffd700 1 --2- 172.21.15.49:0/1934699607 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcfdc02ff60 0x7fcfdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:51.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.239+0000 7fcfe6ffd700 1 -- 172.21.15.49:0/1934699607 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfec0409b0 con 0x7fcff00f6950 2023-11-30T21:45:51.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.239+0000 7fcff584e700 1 --2- 172.21.15.49:0/1934699607 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcfdc02ff60 0x7fcfdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:51.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.240+0000 7fcff584e700 1 --2- 172.21.15.49:0/1934699607 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcfdc02ff60 0x7fcfdc032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fcfe00096a0 tx=0x7fcfe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:51.244 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.244+0000 7fcfe6ffd700 1 -- 172.21.15.49:0/1934699607 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfec0079c0 con 0x7fcff00f6950 2023-11-30T21:45:51.461 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.460+0000 7fcff82b3700 1 -- 172.21.15.49:0/1934699607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcff0107010 con 0x7fcff00f6950 2023-11-30T21:45:51.462 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.461+0000 7fcfe6ffd700 1 -- 172.21.15.49:0/1934699607 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcfec00f300 con 0x7fcff00f6950 2023-11-30T21:45:51.462 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:51.462 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:51.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.462+0000 7fcff82b3700 1 -- 172.21.15.49:0/1934699607 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcfdc02ff60 msgr2=0x7fcfdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:51.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.462+0000 7fcff82b3700 1 --2- 172.21.15.49:0/1934699607 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcfdc02ff60 0x7fcfdc032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fcfe00096a0 tx=0x7fcfe0006b40).stop 2023-11-30T21:45:51.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.462+0000 7fcff82b3700 1 -- 172.21.15.49:0/1934699607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff00f6950 msgr2=0x7fcff01055e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:51.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.462+0000 7fcff82b3700 1 --2- 172.21.15.49:0/1934699607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff00f6950 0x7fcff01055e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fcfec014000 tx=0x7fcfec00b040).stop 2023-11-30T21:45:51.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.462+0000 7fcff82b3700 1 -- 172.21.15.49:0/1934699607 shutdown_connections 2023-11-30T21:45:51.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.462+0000 7fcff82b3700 1 --2- 172.21.15.49:0/1934699607 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcfdc02ff60 0x7fcfdc032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:51.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.462+0000 7fcff82b3700 1 --2- 172.21.15.49:0/1934699607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcff00f6950 0x7fcff01055e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:51.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.462+0000 7fcff82b3700 1 -- 172.21.15.49:0/1934699607 >> 172.21.15.49:0/1934699607 conn(0x7fcff00f0260 msgr2=0x7fcff00f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:51.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.462+0000 7fcff82b3700 1 -- 172.21.15.49:0/1934699607 shutdown_connections 2023-11-30T21:45:51.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:51.462+0000 7fcff82b3700 1 -- 172.21.15.49:0/1934699607 wait complete. 2023-11-30T21:45:51.464 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:51.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:49.472042+0000 mgr.smithi027.brrjcl (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:52.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:52 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:51.461195+0000 mon.smithi027 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.49:0/1934699607' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:52.863 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:52.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:53.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:53 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:51.472331+0000 mgr.smithi027.brrjcl (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:54.095 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.091+0000 7fd07e9b7700 1 -- 172.21.15.49:0/957059806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0780601e0 msgr2=0x7fd0780fb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:54.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.091+0000 7fd07e9b7700 1 --2- 172.21.15.49:0/957059806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0780601e0 0x7fd0780fb320 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd074009fb0 tx=0x7fd074009b30).stop 2023-11-30T21:45:54.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.091+0000 7fd07e9b7700 1 -- 172.21.15.49:0/957059806 shutdown_connections 2023-11-30T21:45:54.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.091+0000 7fd07e9b7700 1 --2- 172.21.15.49:0/957059806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0780601e0 0x7fd0780fb320 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:54.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.091+0000 7fd07e9b7700 1 -- 172.21.15.49:0/957059806 >> 172.21.15.49:0/957059806 conn(0x7fd0780f09a0 msgr2=0x7fd0780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:54.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.091+0000 7fd07e9b7700 1 -- 172.21.15.49:0/957059806 shutdown_connections 2023-11-30T21:45:54.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.091+0000 7fd07e9b7700 1 -- 172.21.15.49:0/957059806 wait complete. 2023-11-30T21:45:54.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.091+0000 7fd07e9b7700 1 Processor -- start 2023-11-30T21:45:54.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.092+0000 7fd07e9b7700 1 -- start start 2023-11-30T21:45:54.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.092+0000 7fd07e9b7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0780601e0 0x7fd0781086f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:54.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.092+0000 7fd07e9b7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd078108be0 con 0x7fd0780601e0 2023-11-30T21:45:54.096 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.092+0000 7fd07d9b5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0780601e0 0x7fd0781086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:54.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.092+0000 7fd07d9b5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0780601e0 0x7fd0781086f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:33156/0 (socket says 172.21.15.49:33156) 2023-11-30T21:45:54.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.092+0000 7fd07d9b5700 1 -- 172.21.15.49:0/3352942138 learned_addr learned my addr 172.21.15.49:0/3352942138 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:54.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.092+0000 7fd07d9b5700 1 -- 172.21.15.49:0/3352942138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd074005040 con 0x7fd0780601e0 2023-11-30T21:45:54.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.092+0000 7fd07d9b5700 1 --2- 172.21.15.49:0/3352942138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0780601e0 0x7fd0781086f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fd074013510 tx=0x7fd07400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:54.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.093+0000 7fd06e7fc700 1 -- 172.21.15.49:0/3352942138 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd074009320 con 0x7fd0780601e0 2023-11-30T21:45:54.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.093+0000 7fd07e9b7700 1 -- 172.21.15.49:0/3352942138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd078108de0 con 0x7fd0780601e0 2023-11-30T21:45:54.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.093+0000 7fd06e7fc700 1 -- 172.21.15.49:0/3352942138 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd074013f00 con 0x7fd0780601e0 2023-11-30T21:45:54.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.093+0000 7fd06e7fc700 1 -- 172.21.15.49:0/3352942138 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd074014d90 con 0x7fd0780601e0 2023-11-30T21:45:54.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.093+0000 7fd07e9b7700 1 -- 172.21.15.49:0/3352942138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd078109270 con 0x7fd0780601e0 2023-11-30T21:45:54.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.093+0000 7fd06e7fc700 1 -- 172.21.15.49:0/3352942138 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0740263a0 con 0x7fd0780601e0 2023-11-30T21:45:54.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.094+0000 7fd06e7fc700 1 --2- 172.21.15.49:0/3352942138 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd06402ffb0 0x7fd064032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:54.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.094+0000 7fd06e7fc700 1 -- 172.21.15.49:0/3352942138 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd074040ff0 con 0x7fd0780601e0 2023-11-30T21:45:54.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.094+0000 7fd07e9b7700 1 -- 172.21.15.49:0/3352942138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd078045ad0 con 0x7fd0780601e0 2023-11-30T21:45:54.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.094+0000 7fd07d1b4700 1 --2- 172.21.15.49:0/3352942138 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd06402ffb0 0x7fd064032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:54.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.094+0000 7fd07d1b4700 1 --2- 172.21.15.49:0/3352942138 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd06402ffb0 0x7fd064032470 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd0680096a0 tx=0x7fd068006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:54.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.099+0000 7fd06e7fc700 1 -- 172.21.15.49:0/3352942138 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd074012920 con 0x7fd0780601e0 2023-11-30T21:45:54.315 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.314+0000 7fd07e9b7700 1 -- 172.21.15.49:0/3352942138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd07810a120 con 0x7fd0780601e0 2023-11-30T21:45:54.315 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.315+0000 7fd06e7fc700 1 -- 172.21.15.49:0/3352942138 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd07402b020 con 0x7fd0780601e0 2023-11-30T21:45:54.316 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:54.316 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:54.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.316+0000 7fd07e9b7700 1 -- 172.21.15.49:0/3352942138 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd06402ffb0 msgr2=0x7fd064032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:54.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.316+0000 7fd07e9b7700 1 --2- 172.21.15.49:0/3352942138 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd06402ffb0 0x7fd064032470 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd0680096a0 tx=0x7fd068006b40).stop 2023-11-30T21:45:54.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.316+0000 7fd07e9b7700 1 -- 172.21.15.49:0/3352942138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0780601e0 msgr2=0x7fd0781086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:54.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.316+0000 7fd07e9b7700 1 --2- 172.21.15.49:0/3352942138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0780601e0 0x7fd0781086f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fd074013510 tx=0x7fd07400b040).stop 2023-11-30T21:45:54.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.316+0000 7fd07e9b7700 1 -- 172.21.15.49:0/3352942138 shutdown_connections 2023-11-30T21:45:54.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.316+0000 7fd07e9b7700 1 --2- 172.21.15.49:0/3352942138 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd06402ffb0 0x7fd064032470 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:54.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.316+0000 7fd07e9b7700 1 --2- 172.21.15.49:0/3352942138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0780601e0 0x7fd0781086f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:54.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.316+0000 7fd07e9b7700 1 -- 172.21.15.49:0/3352942138 >> 172.21.15.49:0/3352942138 conn(0x7fd0780f09a0 msgr2=0x7fd0780f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:54.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.316+0000 7fd07e9b7700 1 -- 172.21.15.49:0/3352942138 shutdown_connections 2023-11-30T21:45:54.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:54.316+0000 7fd07e9b7700 1 -- 172.21.15.49:0/3352942138 wait complete. 2023-11-30T21:45:54.318 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:55.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:53.472686+0000 mgr.smithi027.brrjcl (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:55.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:54.315279+0000 mon.smithi027 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.49:0/3352942138' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:55.730 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:55.730 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:56.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.916+0000 7f44a4825700 1 -- 172.21.15.49:0/3039194160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a0060280 msgr2=0x7f44a00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:56.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.916+0000 7f44a4825700 1 --2- 172.21.15.49:0/3039194160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a0060280 0x7f44a00f90f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f4490009fb0 tx=0x7f4490009b30).stop 2023-11-30T21:45:56.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.916+0000 7f44a4825700 1 -- 172.21.15.49:0/3039194160 shutdown_connections 2023-11-30T21:45:56.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.916+0000 7f44a4825700 1 --2- 172.21.15.49:0/3039194160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a0060280 0x7f44a00f90f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:56.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.916+0000 7f44a4825700 1 -- 172.21.15.49:0/3039194160 >> 172.21.15.49:0/3039194160 conn(0x7f44a00f0b70 msgr2=0x7f44a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:56.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.916+0000 7f44a4825700 1 -- 172.21.15.49:0/3039194160 shutdown_connections 2023-11-30T21:45:56.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.916+0000 7f44a4825700 1 -- 172.21.15.49:0/3039194160 wait complete. 2023-11-30T21:45:56.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.917+0000 7f44a4825700 1 Processor -- start 2023-11-30T21:45:56.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.917+0000 7f44a4825700 1 -- start start 2023-11-30T21:45:56.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.917+0000 7f44a4825700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a0060280 0x7f44a00f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:56.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.917+0000 7f44a4825700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a00f4ff0 con 0x7f44a0060280 2023-11-30T21:45:56.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.917+0000 7f449e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a0060280 0x7f44a00f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:56.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.917+0000 7f449e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a0060280 0x7f44a00f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:33162/0 (socket says 172.21.15.49:33162) 2023-11-30T21:45:56.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.917+0000 7f449e59c700 1 -- 172.21.15.49:0/3130262529 learned_addr learned my addr 172.21.15.49:0/3130262529 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:56.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.918+0000 7f449e59c700 1 -- 172.21.15.49:0/3130262529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4490005040 con 0x7f44a0060280 2023-11-30T21:45:56.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.918+0000 7f449e59c700 1 --2- 172.21.15.49:0/3130262529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a0060280 0x7f44a00f6b70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f4490013510 tx=0x7f449000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:56.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.918+0000 7f4496ffd700 1 -- 172.21.15.49:0/3130262529 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4490009320 con 0x7f44a0060280 2023-11-30T21:45:56.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.918+0000 7f44a4825700 1 -- 172.21.15.49:0/3130262529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a00f5190 con 0x7f44a0060280 2023-11-30T21:45:56.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.918+0000 7f44a4825700 1 -- 172.21.15.49:0/3130262529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a00f5620 con 0x7f44a0060280 2023-11-30T21:45:56.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.918+0000 7f4496ffd700 1 -- 172.21.15.49:0/3130262529 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4490013f00 con 0x7f44a0060280 2023-11-30T21:45:56.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.918+0000 7f4496ffd700 1 -- 172.21.15.49:0/3130262529 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4490014cc0 con 0x7f44a0060280 2023-11-30T21:45:56.922 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.919+0000 7f4496ffd700 1 -- 172.21.15.49:0/3130262529 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4490014420 con 0x7f44a0060280 2023-11-30T21:45:56.922 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.919+0000 7f44a4825700 1 -- 172.21.15.49:0/3130262529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4484004fa0 con 0x7f44a0060280 2023-11-30T21:45:56.922 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.919+0000 7f4496ffd700 1 --2- 172.21.15.49:0/3130262529 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f448c02ff60 0x7f448c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:56.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.919+0000 7f4496ffd700 1 -- 172.21.15.49:0/3130262529 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4490041130 con 0x7f44a0060280 2023-11-30T21:45:56.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.919+0000 7f449dd9b700 1 --2- 172.21.15.49:0/3130262529 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f448c02ff60 0x7f448c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:56.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.920+0000 7f449dd9b700 1 --2- 172.21.15.49:0/3130262529 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f448c02ff60 0x7f448c032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f44880096a0 tx=0x7f4488006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:56.925 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:56.924+0000 7f4496ffd700 1 -- 172.21.15.49:0/3130262529 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44900095f0 con 0x7f44a0060280 2023-11-30T21:45:57.146 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:57.142+0000 7f44a4825700 1 -- 172.21.15.49:0/3130262529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4484005670 con 0x7f44a0060280 2023-11-30T21:45:57.146 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:57.143+0000 7f4496ffd700 1 -- 172.21.15.49:0/3130262529 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f44900095f0 con 0x7f44a0060280 2023-11-30T21:45:57.147 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:45:57.147 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:45:57.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:57.144+0000 7f44a4825700 1 -- 172.21.15.49:0/3130262529 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f448c02ff60 msgr2=0x7f448c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:57.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:57.144+0000 7f44a4825700 1 --2- 172.21.15.49:0/3130262529 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f448c02ff60 0x7f448c032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f44880096a0 tx=0x7f4488006b40).stop 2023-11-30T21:45:57.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:57.144+0000 7f44a4825700 1 -- 172.21.15.49:0/3130262529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a0060280 msgr2=0x7f44a00f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:57.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:57.144+0000 7f44a4825700 1 --2- 172.21.15.49:0/3130262529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a0060280 0x7f44a00f6b70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f4490013510 tx=0x7f449000b040).stop 2023-11-30T21:45:57.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:57.144+0000 7f44a4825700 1 -- 172.21.15.49:0/3130262529 shutdown_connections 2023-11-30T21:45:57.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:57.145+0000 7f44a4825700 1 --2- 172.21.15.49:0/3130262529 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f448c02ff60 0x7f448c032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:57.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:57.145+0000 7f44a4825700 1 --2- 172.21.15.49:0/3130262529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a0060280 0x7f44a00f6b70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:57.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:57.145+0000 7f44a4825700 1 -- 172.21.15.49:0/3130262529 >> 172.21.15.49:0/3130262529 conn(0x7f44a00f0b70 msgr2=0x7f44a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:57.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:57.145+0000 7f44a4825700 1 -- 172.21.15.49:0/3130262529 shutdown_connections 2023-11-30T21:45:57.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:57.145+0000 7f44a4825700 1 -- 172.21.15.49:0/3130262529 wait complete. 2023-11-30T21:45:57.148 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:45:57.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:57 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:55.473101+0000 mgr.smithi027.brrjcl (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:57.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:57 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:45:57.144090+0000 mon.smithi027 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.49:0/3130262529' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:45:58.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:45:58.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:45:59.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:45:59 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:57.473429+0000 mgr.smithi027.brrjcl (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:45:59.911 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.908+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3172035288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e080f5410 msgr2=0x7f0e080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:45:59.911 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.908+0000 7f0e0cfcb700 1 --2- 172.21.15.49:0/3172035288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e080f5410 0x7f0e080f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f0df8009fb0 tx=0x7f0df8009b30).stop 2023-11-30T21:45:59.911 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.908+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3172035288 shutdown_connections 2023-11-30T21:45:59.911 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.908+0000 7f0e0cfcb700 1 --2- 172.21.15.49:0/3172035288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e080f5410 0x7f0e080f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:45:59.911 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.908+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3172035288 >> 172.21.15.49:0/3172035288 conn(0x7f0e080f0b70 msgr2=0x7f0e080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:45:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.908+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3172035288 shutdown_connections 2023-11-30T21:45:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.908+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3172035288 wait complete. 2023-11-30T21:45:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.909+0000 7f0e0cfcb700 1 Processor -- start 2023-11-30T21:45:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.909+0000 7f0e0cfcb700 1 -- start start 2023-11-30T21:45:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.909+0000 7f0e0cfcb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e080f5410 0x7f0e08109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.909+0000 7f0e0cfcb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e08060220 con 0x7f0e080f5410 2023-11-30T21:45:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.909+0000 7f0e0659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e080f5410 0x7f0e08109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.909+0000 7f0e0659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e080f5410 0x7f0e08109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:35998/0 (socket says 172.21.15.49:35998) 2023-11-30T21:45:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.909+0000 7f0e0659c700 1 -- 172.21.15.49:0/3223877531 learned_addr learned my addr 172.21.15.49:0/3223877531 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:45:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.909+0000 7f0e0659c700 1 -- 172.21.15.49:0/3223877531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df8005040 con 0x7f0e080f5410 2023-11-30T21:45:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.909+0000 7f0e0659c700 1 --2- 172.21.15.49:0/3223877531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e080f5410 0x7f0e08109450 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f0df8013510 tx=0x7f0df800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.910+0000 7f0df6ffd700 1 -- 172.21.15.49:0/3223877531 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0df8009320 con 0x7f0e080f5410 2023-11-30T21:45:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.910+0000 7f0df6ffd700 1 -- 172.21.15.49:0/3223877531 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0df8013f00 con 0x7f0e080f5410 2023-11-30T21:45:59.913 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.910+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3223877531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e08060420 con 0x7f0e080f5410 2023-11-30T21:45:59.913 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.910+0000 7f0df6ffd700 1 -- 172.21.15.49:0/3223877531 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0df8014d90 con 0x7f0e080f5410 2023-11-30T21:45:59.913 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.910+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3223877531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e08109b00 con 0x7f0e080f5410 2023-11-30T21:45:59.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.910+0000 7f0df6ffd700 1 -- 172.21.15.49:0/3223877531 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0df80095f0 con 0x7f0e080f5410 2023-11-30T21:45:59.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.911+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3223877531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e08059100 con 0x7f0e080f5410 2023-11-30T21:45:59.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.911+0000 7f0df6ffd700 1 --2- 172.21.15.49:0/3223877531 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0df002ff60 0x7f0df0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:45:59.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.911+0000 7f0df6ffd700 1 -- 172.21.15.49:0/3223877531 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0df80410a0 con 0x7f0e080f5410 2023-11-30T21:45:59.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.911+0000 7f0e05d9b700 1 --2- 172.21.15.49:0/3223877531 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0df002ff60 0x7f0df0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:45:59.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.911+0000 7f0e05d9b700 1 --2- 172.21.15.49:0/3223877531 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0df002ff60 0x7f0df0032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0dfc0096a0 tx=0x7f0dfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:45:59.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:45:59.916+0000 7f0df6ffd700 1 -- 172.21.15.49:0/3223877531 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0df801d7f0 con 0x7f0e080f5410 2023-11-30T21:46:00.137 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:00.135+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3223877531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e08045ad0 con 0x7f0e080f5410 2023-11-30T21:46:00.137 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:00.136+0000 7f0df6ffd700 1 -- 172.21.15.49:0/3223877531 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0df8010230 con 0x7f0e080f5410 2023-11-30T21:46:00.137 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:00.137 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:00.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:00.137+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3223877531 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0df002ff60 msgr2=0x7f0df0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:00.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:00.137+0000 7f0e0cfcb700 1 --2- 172.21.15.49:0/3223877531 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0df002ff60 0x7f0df0032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f0dfc0096a0 tx=0x7f0dfc006b40).stop 2023-11-30T21:46:00.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:00.137+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3223877531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e080f5410 msgr2=0x7f0e08109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:00.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:00.137+0000 7f0e0cfcb700 1 --2- 172.21.15.49:0/3223877531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e080f5410 0x7f0e08109450 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f0df8013510 tx=0x7f0df800b040).stop 2023-11-30T21:46:00.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:00.137+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3223877531 shutdown_connections 2023-11-30T21:46:00.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:00.137+0000 7f0e0cfcb700 1 --2- 172.21.15.49:0/3223877531 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0df002ff60 0x7f0df0032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:00.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:00.137+0000 7f0e0cfcb700 1 --2- 172.21.15.49:0/3223877531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e080f5410 0x7f0e08109450 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:00.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:00.137+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3223877531 >> 172.21.15.49:0/3223877531 conn(0x7f0e080f0b70 msgr2=0x7f0e080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:00.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:00.137+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3223877531 shutdown_connections 2023-11-30T21:46:00.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:00.137+0000 7f0e0cfcb700 1 -- 172.21.15.49:0/3223877531 wait complete. 2023-11-30T21:46:00.140 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:00.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:00 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:00.137225+0000 mon.smithi027 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.49:0/3223877531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:01.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:01 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:45:59.473795+0000 mgr.smithi027.brrjcl (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:01.519 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:01.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:02.782 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.776+0000 7f64aaf02700 1 -- 172.21.15.49:0/1000802764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64a40601e0 msgr2=0x7f64a40f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:02.783 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.776+0000 7f64aaf02700 1 --2- 172.21.15.49:0/1000802764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64a40601e0 0x7f64a40f8f20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f64a0009fb0 tx=0x7f64a0009b30).stop 2023-11-30T21:46:02.783 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.776+0000 7f64aaf02700 1 -- 172.21.15.49:0/1000802764 shutdown_connections 2023-11-30T21:46:02.783 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.776+0000 7f64aaf02700 1 --2- 172.21.15.49:0/1000802764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64a40601e0 0x7f64a40f8f20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:02.784 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.776+0000 7f64aaf02700 1 -- 172.21.15.49:0/1000802764 >> 172.21.15.49:0/1000802764 conn(0x7f64a40f09a0 msgr2=0x7f64a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:02.784 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.776+0000 7f64aaf02700 1 -- 172.21.15.49:0/1000802764 shutdown_connections 2023-11-30T21:46:02.784 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.776+0000 7f64aaf02700 1 -- 172.21.15.49:0/1000802764 wait complete. 2023-11-30T21:46:02.784 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.776+0000 7f64aaf02700 1 Processor -- start 2023-11-30T21:46:02.785 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.777+0000 7f64aaf02700 1 -- start start 2023-11-30T21:46:02.785 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.777+0000 7f64aaf02700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64a40601e0 0x7f64a4108720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:02.785 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.777+0000 7f64aaf02700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a4108c10 con 0x7f64a40601e0 2023-11-30T21:46:02.785 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.777+0000 7f64a9f00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64a40601e0 0x7f64a4108720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:02.786 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.777+0000 7f64a9f00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64a40601e0 0x7f64a4108720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:36018/0 (socket says 172.21.15.49:36018) 2023-11-30T21:46:02.786 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.777+0000 7f64a9f00700 1 -- 172.21.15.49:0/789039808 learned_addr learned my addr 172.21.15.49:0/789039808 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:02.786 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.777+0000 7f64a9f00700 1 -- 172.21.15.49:0/789039808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a0005040 con 0x7f64a40601e0 2023-11-30T21:46:02.787 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.777+0000 7f64a9f00700 1 --2- 172.21.15.49:0/789039808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64a40601e0 0x7f64a4108720 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f64a0013510 tx=0x7f64a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:02.787 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.778+0000 7f649a7fc700 1 -- 172.21.15.49:0/789039808 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64a0009320 con 0x7f64a40601e0 2023-11-30T21:46:02.789 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.778+0000 7f649a7fc700 1 -- 172.21.15.49:0/789039808 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64a0013f00 con 0x7f64a40601e0 2023-11-30T21:46:02.789 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.778+0000 7f64aaf02700 1 -- 172.21.15.49:0/789039808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a4108e10 con 0x7f64a40601e0 2023-11-30T21:46:02.790 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.778+0000 7f649a7fc700 1 -- 172.21.15.49:0/789039808 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64a0014d90 con 0x7f64a40601e0 2023-11-30T21:46:02.790 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.778+0000 7f64aaf02700 1 -- 172.21.15.49:0/789039808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a41092a0 con 0x7f64a40601e0 2023-11-30T21:46:02.790 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.778+0000 7f649a7fc700 1 -- 172.21.15.49:0/789039808 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64a0014420 con 0x7f64a40601e0 2023-11-30T21:46:02.791 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.778+0000 7f64aaf02700 1 -- 172.21.15.49:0/789039808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a4045ad0 con 0x7f64a40601e0 2023-11-30T21:46:02.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.779+0000 7f649a7fc700 1 --2- 172.21.15.49:0/789039808 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f649002ffb0 0x7f6490032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:02.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.779+0000 7f649a7fc700 1 -- 172.21.15.49:0/789039808 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64a0040fd0 con 0x7f64a40601e0 2023-11-30T21:46:02.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.779+0000 7f64a96ff700 1 --2- 172.21.15.49:0/789039808 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f649002ffb0 0x7f6490032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:02.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.779+0000 7f64a96ff700 1 --2- 172.21.15.49:0/789039808 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f649002ffb0 0x7f6490032470 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f64940096a0 tx=0x7f6494006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:02.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:02.783+0000 7f649a7fc700 1 -- 172.21.15.49:0/789039808 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64a006d2b0 con 0x7f64a40601e0 2023-11-30T21:46:03.004 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:03.002+0000 7f64aaf02700 1 -- 172.21.15.49:0/789039808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64a410a150 con 0x7f64a40601e0 2023-11-30T21:46:03.004 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:03.003+0000 7f649a7fc700 1 -- 172.21.15.49:0/789039808 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f64a00095f0 con 0x7f64a40601e0 2023-11-30T21:46:03.004 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:03.005 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:03.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:03.004+0000 7f64aaf02700 1 -- 172.21.15.49:0/789039808 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f649002ffb0 msgr2=0x7f6490032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:03.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:03.004+0000 7f64aaf02700 1 --2- 172.21.15.49:0/789039808 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f649002ffb0 0x7f6490032470 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f64940096a0 tx=0x7f6494006b40).stop 2023-11-30T21:46:03.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:03.004+0000 7f64aaf02700 1 -- 172.21.15.49:0/789039808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64a40601e0 msgr2=0x7f64a4108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:03.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:03.004+0000 7f64aaf02700 1 --2- 172.21.15.49:0/789039808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64a40601e0 0x7f64a4108720 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f64a0013510 tx=0x7f64a000b040).stop 2023-11-30T21:46:03.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:03.004+0000 7f64aaf02700 1 -- 172.21.15.49:0/789039808 shutdown_connections 2023-11-30T21:46:03.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:03.004+0000 7f64aaf02700 1 --2- 172.21.15.49:0/789039808 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f649002ffb0 0x7f6490032470 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:03.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:03.004+0000 7f64aaf02700 1 --2- 172.21.15.49:0/789039808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64a40601e0 0x7f64a4108720 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:03.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:03.004+0000 7f64aaf02700 1 -- 172.21.15.49:0/789039808 >> 172.21.15.49:0/789039808 conn(0x7f64a40f09a0 msgr2=0x7f64a40f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:03.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:03.004+0000 7f64aaf02700 1 -- 172.21.15.49:0/789039808 shutdown_connections 2023-11-30T21:46:03.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:03.004+0000 7f64aaf02700 1 -- 172.21.15.49:0/789039808 wait complete. 2023-11-30T21:46:03.009 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:03.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:03 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:01.474140+0000 mgr.smithi027.brrjcl (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:03.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:03 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:03.004277+0000 mon.smithi027 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.49:0/789039808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:04.395 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:04.395 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:03.584216+0000 mon.smithi027 (mon.0) 146 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/mirror_snapshot_schedule"}]: dispatch 2023-11-30T21:46:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:03.591800+0000 mon.smithi027 (mon.0) 147 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/trash_purge_schedule"}]: dispatch 2023-11-30T21:46:05.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:03.474405+0000 mgr.smithi027.brrjcl (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:05.538 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.537+0000 7ff776c2b700 1 -- 172.21.15.49:0/3576062884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7700f51a0 msgr2=0x7ff7700f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:05.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.537+0000 7ff776c2b700 1 --2- 172.21.15.49:0/3576062884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7700f51a0 0x7ff7700f55c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7ff76c009960 tx=0x7ff76c004d10).stop 2023-11-30T21:46:05.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.537+0000 7ff776c2b700 1 -- 172.21.15.49:0/3576062884 shutdown_connections 2023-11-30T21:46:05.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.537+0000 7ff776c2b700 1 --2- 172.21.15.49:0/3576062884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7700f51a0 0x7ff7700f55c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:05.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.537+0000 7ff776c2b700 1 -- 172.21.15.49:0/3576062884 >> 172.21.15.49:0/3576062884 conn(0x7ff7700f09a0 msgr2=0x7ff7700f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:05.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.537+0000 7ff776c2b700 1 -- 172.21.15.49:0/3576062884 shutdown_connections 2023-11-30T21:46:05.540 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.537+0000 7ff776c2b700 1 -- 172.21.15.49:0/3576062884 wait complete. 2023-11-30T21:46:05.543 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.538+0000 7ff776c2b700 1 Processor -- start 2023-11-30T21:46:05.543 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.538+0000 7ff776c2b700 1 -- start start 2023-11-30T21:46:05.543 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.538+0000 7ff776c2b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7700f51a0 0x7ff7701053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:05.545 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.538+0000 7ff776c2b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7701058a0 con 0x7ff7700f51a0 2023-11-30T21:46:05.545 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.538+0000 7ff775c29700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7700f51a0 0x7ff7701053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:05.545 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.538+0000 7ff775c29700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7700f51a0 0x7ff7701053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:36032/0 (socket says 172.21.15.49:36032) 2023-11-30T21:46:05.545 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.538+0000 7ff775c29700 1 -- 172.21.15.49:0/3047617831 learned_addr learned my addr 172.21.15.49:0/3047617831 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:05.545 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.538+0000 7ff775c29700 1 -- 172.21.15.49:0/3047617831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c009640 con 0x7ff7700f51a0 2023-11-30T21:46:05.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.538+0000 7ff775c29700 1 --2- 172.21.15.49:0/3047617831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7700f51a0 0x7ff7701053b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff76c007bd0 tx=0x7ff76c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:05.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.539+0000 7ff7667fc700 1 -- 172.21.15.49:0/3047617831 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff76c018070 con 0x7ff7700f51a0 2023-11-30T21:46:05.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.539+0000 7ff776c2b700 1 -- 172.21.15.49:0/3047617831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff770105aa0 con 0x7ff7700f51a0 2023-11-30T21:46:05.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.539+0000 7ff7667fc700 1 -- 172.21.15.49:0/3047617831 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff76c00b870 con 0x7ff7700f51a0 2023-11-30T21:46:05.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.539+0000 7ff7667fc700 1 -- 172.21.15.49:0/3047617831 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff76c01d420 con 0x7ff7700f51a0 2023-11-30T21:46:05.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.539+0000 7ff776c2b700 1 -- 172.21.15.49:0/3047617831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff770105f30 con 0x7ff7700f51a0 2023-11-30T21:46:05.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.540+0000 7ff776c2b700 1 -- 172.21.15.49:0/3047617831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff770045ad0 con 0x7ff7700f51a0 2023-11-30T21:46:05.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.540+0000 7ff7667fc700 1 -- 172.21.15.49:0/3047617831 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff76c024060 con 0x7ff7700f51a0 2023-11-30T21:46:05.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.540+0000 7ff7667fc700 1 --2- 172.21.15.49:0/3047617831 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff75c02ff60 0x7ff75c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:05.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.540+0000 7ff7667fc700 1 -- 172.21.15.49:0/3047617831 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff76c041150 con 0x7ff7700f51a0 2023-11-30T21:46:05.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.540+0000 7ff775428700 1 --2- 172.21.15.49:0/3047617831 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff75c02ff60 0x7ff75c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:05.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.540+0000 7ff775428700 1 --2- 172.21.15.49:0/3047617831 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff75c02ff60 0x7ff75c032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ff7600096a0 tx=0x7ff760006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:05.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.544+0000 7ff7667fc700 1 -- 172.21.15.49:0/3047617831 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff76c025520 con 0x7ff7700f51a0 2023-11-30T21:46:05.764 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.762+0000 7ff776c2b700 1 -- 172.21.15.49:0/3047617831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff770106d30 con 0x7ff7700f51a0 2023-11-30T21:46:05.764 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.762+0000 7ff7667fc700 1 -- 172.21.15.49:0/3047617831 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff76c01a2d0 con 0x7ff7700f51a0 2023-11-30T21:46:05.765 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:05.765 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:05.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.764+0000 7ff776c2b700 1 -- 172.21.15.49:0/3047617831 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff75c02ff60 msgr2=0x7ff75c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:05.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.764+0000 7ff776c2b700 1 --2- 172.21.15.49:0/3047617831 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff75c02ff60 0x7ff75c032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ff7600096a0 tx=0x7ff760006b40).stop 2023-11-30T21:46:05.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.764+0000 7ff776c2b700 1 -- 172.21.15.49:0/3047617831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7700f51a0 msgr2=0x7ff7701053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:05.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.764+0000 7ff776c2b700 1 --2- 172.21.15.49:0/3047617831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7700f51a0 0x7ff7701053b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff76c007bd0 tx=0x7ff76c004d10).stop 2023-11-30T21:46:05.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.764+0000 7ff776c2b700 1 -- 172.21.15.49:0/3047617831 shutdown_connections 2023-11-30T21:46:05.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.764+0000 7ff776c2b700 1 --2- 172.21.15.49:0/3047617831 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff75c02ff60 0x7ff75c032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:05.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.764+0000 7ff776c2b700 1 --2- 172.21.15.49:0/3047617831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7700f51a0 0x7ff7701053b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:05.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.764+0000 7ff776c2b700 1 -- 172.21.15.49:0/3047617831 >> 172.21.15.49:0/3047617831 conn(0x7ff7700f09a0 msgr2=0x7ff7700f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:05.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.764+0000 7ff776c2b700 1 -- 172.21.15.49:0/3047617831 shutdown_connections 2023-11-30T21:46:05.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:05.764+0000 7ff776c2b700 1 -- 172.21.15.49:0/3047617831 wait complete. 2023-11-30T21:46:05.767 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:06.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:06 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:05.764060+0000 mon.smithi027 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.49:0/3047617831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:07.177 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:07.178 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:07.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:07 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:05.474662+0000 mgr.smithi027.brrjcl (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:08.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.319+0000 7f4ead52d700 1 -- 172.21.15.49:0/4010947816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ea80f62a0 msgr2=0x7f4ea80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:08.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.319+0000 7f4ead52d700 1 --2- 172.21.15.49:0/4010947816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ea80f62a0 0x7f4ea80f66c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4e98009fb0 tx=0x7f4e98009b30).stop 2023-11-30T21:46:08.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.319+0000 7f4ead52d700 1 -- 172.21.15.49:0/4010947816 shutdown_connections 2023-11-30T21:46:08.321 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.319+0000 7f4ead52d700 1 --2- 172.21.15.49:0/4010947816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ea80f62a0 0x7f4ea80f66c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:08.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.319+0000 7f4ead52d700 1 -- 172.21.15.49:0/4010947816 >> 172.21.15.49:0/4010947816 conn(0x7f4ea80f1aa0 msgr2=0x7f4ea80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:08.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.319+0000 7f4ead52d700 1 -- 172.21.15.49:0/4010947816 shutdown_connections 2023-11-30T21:46:08.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.319+0000 7f4ead52d700 1 -- 172.21.15.49:0/4010947816 wait complete. 2023-11-30T21:46:08.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.320+0000 7f4ead52d700 1 Processor -- start 2023-11-30T21:46:08.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.320+0000 7f4ead52d700 1 -- start start 2023-11-30T21:46:08.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.320+0000 7f4ead52d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ea80f62a0 0x7f4ea81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:08.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.320+0000 7f4ead52d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea81069a0 con 0x7f4ea80f62a0 2023-11-30T21:46:08.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.320+0000 7f4ea7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ea80f62a0 0x7f4ea81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:08.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.320+0000 7f4ea7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ea80f62a0 0x7f4ea81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:36044/0 (socket says 172.21.15.49:36044) 2023-11-30T21:46:08.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.320+0000 7f4ea7fff700 1 -- 172.21.15.49:0/1560060321 learned_addr learned my addr 172.21.15.49:0/1560060321 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:08.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.320+0000 7f4ea7fff700 1 -- 172.21.15.49:0/1560060321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e98005040 con 0x7f4ea80f62a0 2023-11-30T21:46:08.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.321+0000 7f4ea7fff700 1 --2- 172.21.15.49:0/1560060321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ea80f62a0 0x7f4ea81064b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f4e98013510 tx=0x7f4e9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:08.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.321+0000 7f4ea4ff9700 1 -- 172.21.15.49:0/1560060321 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e98009320 con 0x7f4ea80f62a0 2023-11-30T21:46:08.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.321+0000 7f4ea4ff9700 1 -- 172.21.15.49:0/1560060321 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e98013f00 con 0x7f4ea80f62a0 2023-11-30T21:46:08.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.321+0000 7f4ead52d700 1 -- 172.21.15.49:0/1560060321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea8106ba0 con 0x7f4ea80f62a0 2023-11-30T21:46:08.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.321+0000 7f4ea4ff9700 1 -- 172.21.15.49:0/1560060321 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e98014dc0 con 0x7f4ea80f62a0 2023-11-30T21:46:08.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.321+0000 7f4ead52d700 1 -- 172.21.15.49:0/1560060321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea8107030 con 0x7f4ea80f62a0 2023-11-30T21:46:08.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.322+0000 7f4ea4ff9700 1 -- 172.21.15.49:0/1560060321 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e980263a0 con 0x7f4ea80f62a0 2023-11-30T21:46:08.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.322+0000 7f4ea4ff9700 1 --2- 172.21.15.49:0/1560060321 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4e9402ffb0 0x7f4e94032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:08.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.322+0000 7f4ea4ff9700 1 -- 172.21.15.49:0/1560060321 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e98041000 con 0x7f4ea80f62a0 2023-11-30T21:46:08.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.322+0000 7f4ead52d700 1 -- 172.21.15.49:0/1560060321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea8045ad0 con 0x7f4ea80f62a0 2023-11-30T21:46:08.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.322+0000 7f4ea77fe700 1 --2- 172.21.15.49:0/1560060321 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4e9402ffb0 0x7f4e94032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:08.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.322+0000 7f4ea77fe700 1 --2- 172.21.15.49:0/1560060321 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4e9402ffb0 0x7f4e94032470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f4e900096a0 tx=0x7f4e90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:08.328 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.327+0000 7f4ea4ff9700 1 -- 172.21.15.49:0/1560060321 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e98012920 con 0x7f4ea80f62a0 2023-11-30T21:46:08.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.545+0000 7f4ead52d700 1 -- 172.21.15.49:0/1560060321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ea8107d50 con 0x7f4ea80f62a0 2023-11-30T21:46:08.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.546+0000 7f4ea4ff9700 1 -- 172.21.15.49:0/1560060321 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4e9802b020 con 0x7f4ea80f62a0 2023-11-30T21:46:08.547 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:08.547 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:08.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.547+0000 7f4ead52d700 1 -- 172.21.15.49:0/1560060321 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4e9402ffb0 msgr2=0x7f4e94032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:08.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.547+0000 7f4ead52d700 1 --2- 172.21.15.49:0/1560060321 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4e9402ffb0 0x7f4e94032470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f4e900096a0 tx=0x7f4e90006b40).stop 2023-11-30T21:46:08.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.547+0000 7f4ead52d700 1 -- 172.21.15.49:0/1560060321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ea80f62a0 msgr2=0x7f4ea81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:08.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.547+0000 7f4ead52d700 1 --2- 172.21.15.49:0/1560060321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ea80f62a0 0x7f4ea81064b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f4e98013510 tx=0x7f4e9800b040).stop 2023-11-30T21:46:08.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.547+0000 7f4ead52d700 1 -- 172.21.15.49:0/1560060321 shutdown_connections 2023-11-30T21:46:08.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.547+0000 7f4ead52d700 1 --2- 172.21.15.49:0/1560060321 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4e9402ffb0 0x7f4e94032470 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:08.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.547+0000 7f4ead52d700 1 --2- 172.21.15.49:0/1560060321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ea80f62a0 0x7f4ea81064b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:08.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.547+0000 7f4ead52d700 1 -- 172.21.15.49:0/1560060321 >> 172.21.15.49:0/1560060321 conn(0x7f4ea80f1aa0 msgr2=0x7f4ea80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:08.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.547+0000 7f4ead52d700 1 -- 172.21.15.49:0/1560060321 shutdown_connections 2023-11-30T21:46:08.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:08.547+0000 7f4ead52d700 1 -- 172.21.15.49:0/1560060321 wait complete. 2023-11-30T21:46:08.550 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:09.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:09 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:07.474995+0000 mgr.smithi027.brrjcl (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:09.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:09 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:08.547195+0000 mon.smithi027 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.49:0/1560060321' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:09.946 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:09.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:11.115 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.113+0000 7f714e103700 1 -- 172.21.15.49:0/790590632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148060160 msgr2=0x7f71480fb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:11.115 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.113+0000 7f714e103700 1 --2- 172.21.15.49:0/790590632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148060160 0x7f71480fb3a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f7144009fb0 tx=0x7f7144009b30).stop 2023-11-30T21:46:11.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.113+0000 7f714e103700 1 -- 172.21.15.49:0/790590632 shutdown_connections 2023-11-30T21:46:11.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.113+0000 7f714e103700 1 --2- 172.21.15.49:0/790590632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148060160 0x7f71480fb3a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:11.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.113+0000 7f714e103700 1 -- 172.21.15.49:0/790590632 >> 172.21.15.49:0/790590632 conn(0x7f71480f09a0 msgr2=0x7f71480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:11.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.113+0000 7f714e103700 1 -- 172.21.15.49:0/790590632 shutdown_connections 2023-11-30T21:46:11.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.113+0000 7f714e103700 1 -- 172.21.15.49:0/790590632 wait complete. 2023-11-30T21:46:11.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.113+0000 7f714e103700 1 Processor -- start 2023-11-30T21:46:11.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.113+0000 7f714e103700 1 -- start start 2023-11-30T21:46:11.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.113+0000 7f714e103700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148060160 0x7f71481086f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:11.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.113+0000 7f714e103700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7148108be0 con 0x7f7148060160 2023-11-30T21:46:11.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.114+0000 7f714d101700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148060160 0x7f71481086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:11.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.114+0000 7f714d101700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148060160 0x7f71481086f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:40624/0 (socket says 172.21.15.49:40624) 2023-11-30T21:46:11.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.114+0000 7f714d101700 1 -- 172.21.15.49:0/3518137498 learned_addr learned my addr 172.21.15.49:0/3518137498 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:11.117 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.114+0000 7f714d101700 1 -- 172.21.15.49:0/3518137498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7144005040 con 0x7f7148060160 2023-11-30T21:46:11.117 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.114+0000 7f714d101700 1 --2- 172.21.15.49:0/3518137498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148060160 0x7f71481086f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7144013510 tx=0x7f714400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:11.118 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.114+0000 7f713dffb700 1 -- 172.21.15.49:0/3518137498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7144009320 con 0x7f7148060160 2023-11-30T21:46:11.118 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.114+0000 7f714e103700 1 -- 172.21.15.49:0/3518137498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7148108de0 con 0x7f7148060160 2023-11-30T21:46:11.118 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.114+0000 7f713dffb700 1 -- 172.21.15.49:0/3518137498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7144013f00 con 0x7f7148060160 2023-11-30T21:46:11.118 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.114+0000 7f713dffb700 1 -- 172.21.15.49:0/3518137498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7144014d90 con 0x7f7148060160 2023-11-30T21:46:11.118 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.115+0000 7f714e103700 1 -- 172.21.15.49:0/3518137498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7148109270 con 0x7f7148060160 2023-11-30T21:46:11.118 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.115+0000 7f713dffb700 1 -- 172.21.15.49:0/3518137498 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71440095f0 con 0x7f7148060160 2023-11-30T21:46:11.119 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.115+0000 7f714e103700 1 -- 172.21.15.49:0/3518137498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7148045ad0 con 0x7f7148060160 2023-11-30T21:46:11.119 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.116+0000 7f713dffb700 1 --2- 172.21.15.49:0/3518137498 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f713402ff60 0x7f7134032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:11.119 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.116+0000 7f713dffb700 1 -- 172.21.15.49:0/3518137498 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7144041000 con 0x7f7148060160 2023-11-30T21:46:11.119 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.116+0000 7f714c900700 1 --2- 172.21.15.49:0/3518137498 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f713402ff60 0x7f7134032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:11.119 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.116+0000 7f714c900700 1 --2- 172.21.15.49:0/3518137498 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f713402ff60 0x7f7134032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f71380096a0 tx=0x7f7138006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:11.122 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.120+0000 7f713dffb700 1 -- 172.21.15.49:0/3518137498 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7144014420 con 0x7f7148060160 2023-11-30T21:46:11.341 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.339+0000 7f714e103700 1 -- 172.21.15.49:0/3518137498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f714810a120 con 0x7f7148060160 2023-11-30T21:46:11.341 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.340+0000 7f713dffb700 1 -- 172.21.15.49:0/3518137498 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f714401d7f0 con 0x7f7148060160 2023-11-30T21:46:11.342 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:11.342 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:11.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.341+0000 7f714e103700 1 -- 172.21.15.49:0/3518137498 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f713402ff60 msgr2=0x7f7134032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:11.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.341+0000 7f714e103700 1 --2- 172.21.15.49:0/3518137498 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f713402ff60 0x7f7134032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f71380096a0 tx=0x7f7138006b40).stop 2023-11-30T21:46:11.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.341+0000 7f714e103700 1 -- 172.21.15.49:0/3518137498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148060160 msgr2=0x7f71481086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:11.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.341+0000 7f714e103700 1 --2- 172.21.15.49:0/3518137498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148060160 0x7f71481086f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7144013510 tx=0x7f714400b040).stop 2023-11-30T21:46:11.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.341+0000 7f714e103700 1 -- 172.21.15.49:0/3518137498 shutdown_connections 2023-11-30T21:46:11.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.341+0000 7f714e103700 1 --2- 172.21.15.49:0/3518137498 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f713402ff60 0x7f7134032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:11.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.342+0000 7f714e103700 1 --2- 172.21.15.49:0/3518137498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7148060160 0x7f71481086f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:11.343 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.342+0000 7f714e103700 1 -- 172.21.15.49:0/3518137498 >> 172.21.15.49:0/3518137498 conn(0x7f71480f09a0 msgr2=0x7f71480f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:11.344 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.342+0000 7f714e103700 1 -- 172.21.15.49:0/3518137498 shutdown_connections 2023-11-30T21:46:11.344 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:11.342+0000 7f714e103700 1 -- 172.21.15.49:0/3518137498 wait complete. 2023-11-30T21:46:11.344 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:11.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:09.475330+0000 mgr.smithi027.brrjcl (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:12.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:12 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:11.341469+0000 mon.smithi027 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.49:0/3518137498' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:12.792 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:12.793 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:13.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:13 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:11.475670+0000 mgr.smithi027.brrjcl (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:13.999 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.997+0000 7f7457ba9700 1 -- 172.21.15.49:0/1947655911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74500f5410 msgr2=0x7f74500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:13.999 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.997+0000 7f7457ba9700 1 --2- 172.21.15.49:0/1947655911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74500f5410 0x7f74500f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f7440009fb0 tx=0x7f7440009b30).stop 2023-11-30T21:46:14.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.997+0000 7f7457ba9700 1 -- 172.21.15.49:0/1947655911 shutdown_connections 2023-11-30T21:46:14.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.997+0000 7f7457ba9700 1 --2- 172.21.15.49:0/1947655911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74500f5410 0x7f74500f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:14.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.997+0000 7f7457ba9700 1 -- 172.21.15.49:0/1947655911 >> 172.21.15.49:0/1947655911 conn(0x7f74500f0b70 msgr2=0x7f74500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:14.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.997+0000 7f7457ba9700 1 -- 172.21.15.49:0/1947655911 shutdown_connections 2023-11-30T21:46:14.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.997+0000 7f7457ba9700 1 -- 172.21.15.49:0/1947655911 wait complete. 2023-11-30T21:46:14.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.998+0000 7f7457ba9700 1 Processor -- start 2023-11-30T21:46:14.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.998+0000 7f7457ba9700 1 -- start start 2023-11-30T21:46:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.998+0000 7f7457ba9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74500f5410 0x7f74501093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.998+0000 7f7457ba9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7450060280 con 0x7f74500f5410 2023-11-30T21:46:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.998+0000 7f7455945700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74500f5410 0x7f74501093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.998+0000 7f7455945700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74500f5410 0x7f74501093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:40642/0 (socket says 172.21.15.49:40642) 2023-11-30T21:46:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.998+0000 7f7455945700 1 -- 172.21.15.49:0/3961156902 learned_addr learned my addr 172.21.15.49:0/3961156902 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.999+0000 7f7455945700 1 -- 172.21.15.49:0/3961156902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7440005040 con 0x7f74500f5410 2023-11-30T21:46:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.999+0000 7f7455945700 1 --2- 172.21.15.49:0/3961156902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74500f5410 0x7f74501093f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f7440014000 tx=0x7f744000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.999+0000 7f74467fc700 1 -- 172.21.15.49:0/3961156902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74400095f0 con 0x7f74500f5410 2023-11-30T21:46:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.999+0000 7f7457ba9700 1 -- 172.21.15.49:0/3961156902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7450060480 con 0x7f74500f5410 2023-11-30T21:46:14.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.999+0000 7f74467fc700 1 -- 172.21.15.49:0/3961156902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7440007e70 con 0x7f74500f5410 2023-11-30T21:46:14.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.999+0000 7f74467fc700 1 -- 172.21.15.49:0/3961156902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7440014ea0 con 0x7f74500f5410 2023-11-30T21:46:14.005 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:13.999+0000 7f7457ba9700 1 -- 172.21.15.49:0/3961156902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7450109b20 con 0x7f74500f5410 2023-11-30T21:46:14.005 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.000+0000 7f74467fc700 1 -- 172.21.15.49:0/3961156902 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7440025070 con 0x7f74500f5410 2023-11-30T21:46:14.005 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.000+0000 7f7457ba9700 1 -- 172.21.15.49:0/3961156902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7450045ad0 con 0x7f74500f5410 2023-11-30T21:46:14.005 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.000+0000 7f74467fc700 1 --2- 172.21.15.49:0/3961156902 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f743c02ff60 0x7f743c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:14.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.000+0000 7f74467fc700 1 -- 172.21.15.49:0/3961156902 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74400409e0 con 0x7f74500f5410 2023-11-30T21:46:14.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.000+0000 7f7455144700 1 --2- 172.21.15.49:0/3961156902 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f743c02ff60 0x7f743c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:14.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.001+0000 7f7455144700 1 --2- 172.21.15.49:0/3961156902 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f743c02ff60 0x7f743c032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f744c0096a0 tx=0x7f744c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:14.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.005+0000 7f74467fc700 1 -- 172.21.15.49:0/3961156902 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f744000f4e0 con 0x7f74500f5410 2023-11-30T21:46:14.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.224+0000 7f7457ba9700 1 -- 172.21.15.49:0/3961156902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f745010a340 con 0x7f74500f5410 2023-11-30T21:46:14.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.225+0000 7f74467fc700 1 -- 172.21.15.49:0/3961156902 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f744000f300 con 0x7f74500f5410 2023-11-30T21:46:14.227 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:14.227 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:14.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.227+0000 7f7457ba9700 1 -- 172.21.15.49:0/3961156902 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f743c02ff60 msgr2=0x7f743c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:14.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.227+0000 7f7457ba9700 1 --2- 172.21.15.49:0/3961156902 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f743c02ff60 0x7f743c032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f744c0096a0 tx=0x7f744c006b40).stop 2023-11-30T21:46:14.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.227+0000 7f7457ba9700 1 -- 172.21.15.49:0/3961156902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74500f5410 msgr2=0x7f74501093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:14.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.227+0000 7f7457ba9700 1 --2- 172.21.15.49:0/3961156902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74500f5410 0x7f74501093f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f7440014000 tx=0x7f744000b040).stop 2023-11-30T21:46:14.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.227+0000 7f7457ba9700 1 -- 172.21.15.49:0/3961156902 shutdown_connections 2023-11-30T21:46:14.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.227+0000 7f7457ba9700 1 --2- 172.21.15.49:0/3961156902 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f743c02ff60 0x7f743c032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:14.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.227+0000 7f7457ba9700 1 --2- 172.21.15.49:0/3961156902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74500f5410 0x7f74501093f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:14.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.227+0000 7f7457ba9700 1 -- 172.21.15.49:0/3961156902 >> 172.21.15.49:0/3961156902 conn(0x7f74500f0b70 msgr2=0x7f74500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:14.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.227+0000 7f7457ba9700 1 -- 172.21.15.49:0/3961156902 shutdown_connections 2023-11-30T21:46:14.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:14.227+0000 7f7457ba9700 1 -- 172.21.15.49:0/3961156902 wait complete. 2023-11-30T21:46:14.229 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:15.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:15 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:13.476024+0000 mgr.smithi027.brrjcl (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:15.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:15 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:14.226629+0000 mon.smithi027 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.49:0/3961156902' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:15.670 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:15.670 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:16.848 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.846+0000 7f39117e6700 1 -- 172.21.15.49:0/4141740851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0f78e0 msgr2=0x7f390c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:16.848 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.846+0000 7f39117e6700 1 --2- 172.21.15.49:0/4141740851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0f78e0 0x7f390c0f7d00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f38fc009fb0 tx=0x7f38fc009b30).stop 2023-11-30T21:46:16.848 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.846+0000 7f39117e6700 1 -- 172.21.15.49:0/4141740851 shutdown_connections 2023-11-30T21:46:16.848 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.846+0000 7f39117e6700 1 --2- 172.21.15.49:0/4141740851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0f78e0 0x7f390c0f7d00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:16.848 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.846+0000 7f39117e6700 1 -- 172.21.15.49:0/4141740851 >> 172.21.15.49:0/4141740851 conn(0x7f390c0f0120 msgr2=0x7f390c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:16.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.846+0000 7f39117e6700 1 -- 172.21.15.49:0/4141740851 shutdown_connections 2023-11-30T21:46:16.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.846+0000 7f39117e6700 1 -- 172.21.15.49:0/4141740851 wait complete. 2023-11-30T21:46:16.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.846+0000 7f39117e6700 1 Processor -- start 2023-11-30T21:46:16.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.847+0000 7f39117e6700 1 -- start start 2023-11-30T21:46:16.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.847+0000 7f39117e6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0f78e0 0x7f390c1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:16.849 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.847+0000 7f39117e6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c1069d0 con 0x7f390c0f78e0 2023-11-30T21:46:16.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.847+0000 7f390bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0f78e0 0x7f390c1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:16.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.847+0000 7f390bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0f78e0 0x7f390c1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:40662/0 (socket says 172.21.15.49:40662) 2023-11-30T21:46:16.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.847+0000 7f390bfff700 1 -- 172.21.15.49:0/3938713236 learned_addr learned my addr 172.21.15.49:0/3938713236 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:16.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.848+0000 7f390bfff700 1 -- 172.21.15.49:0/3938713236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc005040 con 0x7f390c0f78e0 2023-11-30T21:46:16.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.848+0000 7f390bfff700 1 --2- 172.21.15.49:0/3938713236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0f78e0 0x7f390c1064e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f38fc013510 tx=0x7f38fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:16.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.848+0000 7f3908ff9700 1 -- 172.21.15.49:0/3938713236 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38fc009320 con 0x7f390c0f78e0 2023-11-30T21:46:16.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.848+0000 7f39117e6700 1 -- 172.21.15.49:0/3938713236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390c106bd0 con 0x7f390c0f78e0 2023-11-30T21:46:16.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.848+0000 7f3908ff9700 1 -- 172.21.15.49:0/3938713236 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38fc013f00 con 0x7f390c0f78e0 2023-11-30T21:46:16.851 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.848+0000 7f3908ff9700 1 -- 172.21.15.49:0/3938713236 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38fc014dc0 con 0x7f390c0f78e0 2023-11-30T21:46:16.851 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.848+0000 7f39117e6700 1 -- 172.21.15.49:0/3938713236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390c107060 con 0x7f390c0f78e0 2023-11-30T21:46:16.851 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.849+0000 7f3908ff9700 1 -- 172.21.15.49:0/3938713236 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38fc01d3d0 con 0x7f390c0f78e0 2023-11-30T21:46:16.851 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.849+0000 7f39117e6700 1 -- 172.21.15.49:0/3938713236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390c045ad0 con 0x7f390c0f78e0 2023-11-30T21:46:16.852 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.849+0000 7f3908ff9700 1 --2- 172.21.15.49:0/3938713236 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38f402ff60 0x7f38f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:16.852 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.849+0000 7f3908ff9700 1 -- 172.21.15.49:0/3938713236 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38fc041070 con 0x7f390c0f78e0 2023-11-30T21:46:16.852 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.849+0000 7f390b7fe700 1 --2- 172.21.15.49:0/3938713236 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38f402ff60 0x7f38f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:16.852 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.850+0000 7f390b7fe700 1 --2- 172.21.15.49:0/3938713236 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38f402ff60 0x7f38f4032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f39000096a0 tx=0x7f3900006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:16.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:16.854+0000 7f3908ff9700 1 -- 172.21.15.49:0/3938713236 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38fc01cb10 con 0x7f390c0f78e0 2023-11-30T21:46:17.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:17.076+0000 7f39117e6700 1 -- 172.21.15.49:0/3938713236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f390c107eb0 con 0x7f390c0f78e0 2023-11-30T21:46:17.137 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:17.078+0000 7f3908ff9700 1 -- 172.21.15.49:0/3938713236 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f38fc01cb10 con 0x7f390c0f78e0 2023-11-30T21:46:17.138 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:17.138 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:17.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:17.078+0000 7f39117e6700 1 -- 172.21.15.49:0/3938713236 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38f402ff60 msgr2=0x7f38f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:17.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:17.078+0000 7f39117e6700 1 --2- 172.21.15.49:0/3938713236 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38f402ff60 0x7f38f4032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f39000096a0 tx=0x7f3900006b40).stop 2023-11-30T21:46:17.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:17.078+0000 7f39117e6700 1 -- 172.21.15.49:0/3938713236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0f78e0 msgr2=0x7f390c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:17.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:17.078+0000 7f39117e6700 1 --2- 172.21.15.49:0/3938713236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0f78e0 0x7f390c1064e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f38fc013510 tx=0x7f38fc00b040).stop 2023-11-30T21:46:17.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:17.078+0000 7f39117e6700 1 -- 172.21.15.49:0/3938713236 shutdown_connections 2023-11-30T21:46:17.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:17.078+0000 7f39117e6700 1 --2- 172.21.15.49:0/3938713236 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38f402ff60 0x7f38f4032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:17.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:17.078+0000 7f39117e6700 1 --2- 172.21.15.49:0/3938713236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0f78e0 0x7f390c1064e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:17.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:17.078+0000 7f39117e6700 1 -- 172.21.15.49:0/3938713236 >> 172.21.15.49:0/3938713236 conn(0x7f390c0f0120 msgr2=0x7f390c0f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:17.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:17.078+0000 7f39117e6700 1 -- 172.21.15.49:0/3938713236 shutdown_connections 2023-11-30T21:46:17.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:17.079+0000 7f39117e6700 1 -- 172.21.15.49:0/3938713236 wait complete. 2023-11-30T21:46:17.140 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:17.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:15.476304+0000 mgr.smithi027.brrjcl (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:17.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:17.078291+0000 mon.smithi027 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.49:0/3938713236' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:18.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:18.497 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:19.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:17.476627+0000 mgr.smithi027.brrjcl (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:19.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.694+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/2551834929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00f5410 msgr2=0x7f5ec00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:19.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.694+0000 7f5ec5ffe700 1 --2- 172.21.15.49:0/2551834929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00f5410 0x7f5ec00f5830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f5eb0009fb0 tx=0x7f5eb0009b30).stop 2023-11-30T21:46:19.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.694+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/2551834929 shutdown_connections 2023-11-30T21:46:19.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.694+0000 7f5ec5ffe700 1 --2- 172.21.15.49:0/2551834929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00f5410 0x7f5ec00f5830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:19.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.694+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/2551834929 >> 172.21.15.49:0/2551834929 conn(0x7f5ec00f0b70 msgr2=0x7f5ec00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:19.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.694+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/2551834929 shutdown_connections 2023-11-30T21:46:19.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.694+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/2551834929 wait complete. 2023-11-30T21:46:19.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.694+0000 7f5ec5ffe700 1 Processor -- start 2023-11-30T21:46:19.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.694+0000 7f5ec5ffe700 1 -- start start 2023-11-30T21:46:19.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.695+0000 7f5ec5ffe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00f5410 0x7f5ec0108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:19.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.695+0000 7f5ec5ffe700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec0108e00 con 0x7f5ec00f5410 2023-11-30T21:46:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.695+0000 7f5ebf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00f5410 0x7f5ec0108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.695+0000 7f5ebf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00f5410 0x7f5ec0108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:49744/0 (socket says 172.21.15.49:49744) 2023-11-30T21:46:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.695+0000 7f5ebf7fe700 1 -- 172.21.15.49:0/4287381669 learned_addr learned my addr 172.21.15.49:0/4287381669 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.695+0000 7f5ebf7fe700 1 -- 172.21.15.49:0/4287381669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb0005040 con 0x7f5ec00f5410 2023-11-30T21:46:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.695+0000 7f5ebf7fe700 1 --2- 172.21.15.49:0/4287381669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00f5410 0x7f5ec0108910 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f5eb0013510 tx=0x7f5eb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.696+0000 7f5ea7fff700 1 -- 172.21.15.49:0/4287381669 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5eb0009320 con 0x7f5ec00f5410 2023-11-30T21:46:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.696+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/4287381669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec0109000 con 0x7f5ec00f5410 2023-11-30T21:46:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.696+0000 7f5ea7fff700 1 -- 172.21.15.49:0/4287381669 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5eb0013f00 con 0x7f5ec00f5410 2023-11-30T21:46:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.696+0000 7f5ea7fff700 1 -- 172.21.15.49:0/4287381669 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5eb0014d90 con 0x7f5ec00f5410 2023-11-30T21:46:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.696+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/4287381669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec0109490 con 0x7f5ec00f5410 2023-11-30T21:46:19.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.696+0000 7f5ea7fff700 1 -- 172.21.15.49:0/4287381669 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5eb00263a0 con 0x7f5ec00f5410 2023-11-30T21:46:19.699 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.697+0000 7f5ea7fff700 1 --2- 172.21.15.49:0/4287381669 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5ea802ff60 0x7f5ea8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:19.699 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.697+0000 7f5ea7fff700 1 -- 172.21.15.49:0/4287381669 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5eb0041010 con 0x7f5ec00f5410 2023-11-30T21:46:19.699 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.697+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/4287381669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec0045ad0 con 0x7f5ec00f5410 2023-11-30T21:46:19.699 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.697+0000 7f5ebeffd700 1 --2- 172.21.15.49:0/4287381669 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5ea802ff60 0x7f5ea8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:19.699 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.697+0000 7f5ebeffd700 1 --2- 172.21.15.49:0/4287381669 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5ea802ff60 0x7f5ea8032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f5eb40096a0 tx=0x7f5eb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:19.704 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.702+0000 7f5ea7fff700 1 -- 172.21.15.49:0/4287381669 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5eb0012920 con 0x7f5ec00f5410 2023-11-30T21:46:19.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.922+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/4287381669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ec00faf50 con 0x7f5ec00f5410 2023-11-30T21:46:19.923 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.923+0000 7f5ea7fff700 1 -- 172.21.15.49:0/4287381669 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5eb002b020 con 0x7f5ec00f5410 2023-11-30T21:46:19.924 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:19.924 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:19.925 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.925+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/4287381669 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5ea802ff60 msgr2=0x7f5ea8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:19.925 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.925+0000 7f5ec5ffe700 1 --2- 172.21.15.49:0/4287381669 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5ea802ff60 0x7f5ea8032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f5eb40096a0 tx=0x7f5eb4006b40).stop 2023-11-30T21:46:19.925 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.925+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/4287381669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00f5410 msgr2=0x7f5ec0108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:19.925 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.925+0000 7f5ec5ffe700 1 --2- 172.21.15.49:0/4287381669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00f5410 0x7f5ec0108910 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f5eb0013510 tx=0x7f5eb000b040).stop 2023-11-30T21:46:19.926 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.925+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/4287381669 shutdown_connections 2023-11-30T21:46:19.926 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.925+0000 7f5ec5ffe700 1 --2- 172.21.15.49:0/4287381669 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5ea802ff60 0x7f5ea8032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:19.926 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.925+0000 7f5ec5ffe700 1 --2- 172.21.15.49:0/4287381669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec00f5410 0x7f5ec0108910 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:19.926 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.925+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/4287381669 >> 172.21.15.49:0/4287381669 conn(0x7f5ec00f0b70 msgr2=0x7f5ec005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:19.926 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.925+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/4287381669 shutdown_connections 2023-11-30T21:46:19.926 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:19.925+0000 7f5ec5ffe700 1 -- 172.21.15.49:0/4287381669 wait complete. 2023-11-30T21:46:19.926 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:20.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:19.923336+0000 mon.smithi027 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.49:0/4287381669' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:21.301 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:21.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:21.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:21 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:19.477034+0000 mgr.smithi027.brrjcl (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:22.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.469+0000 7f967e062700 1 -- 172.21.15.49:0/3323710691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96780f67c0 msgr2=0x7f96780f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:22.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.469+0000 7f967e062700 1 --2- 172.21.15.49:0/3323710691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96780f67c0 0x7f96780f6be0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9668009fb0 tx=0x7f9668009b30).stop 2023-11-30T21:46:22.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.469+0000 7f967e062700 1 -- 172.21.15.49:0/3323710691 shutdown_connections 2023-11-30T21:46:22.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.469+0000 7f967e062700 1 --2- 172.21.15.49:0/3323710691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96780f67c0 0x7f96780f6be0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:22.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.469+0000 7f967e062700 1 -- 172.21.15.49:0/3323710691 >> 172.21.15.49:0/3323710691 conn(0x7f96780f00f0 msgr2=0x7f96780f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:22.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.469+0000 7f967e062700 1 -- 172.21.15.49:0/3323710691 shutdown_connections 2023-11-30T21:46:22.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.469+0000 7f967e062700 1 -- 172.21.15.49:0/3323710691 wait complete. 2023-11-30T21:46:22.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.470+0000 7f967e062700 1 Processor -- start 2023-11-30T21:46:22.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.470+0000 7f967e062700 1 -- start start 2023-11-30T21:46:22.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.470+0000 7f967e062700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96780f67c0 0x7f9678106560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:22.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.470+0000 7f967e062700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9678106a50 con 0x7f96780f67c0 2023-11-30T21:46:22.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.470+0000 7f96777fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96780f67c0 0x7f9678106560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:22.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.470+0000 7f96777fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96780f67c0 0x7f9678106560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:49760/0 (socket says 172.21.15.49:49760) 2023-11-30T21:46:22.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.470+0000 7f96777fe700 1 -- 172.21.15.49:0/2624336034 learned_addr learned my addr 172.21.15.49:0/2624336034 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:22.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.471+0000 7f96777fe700 1 -- 172.21.15.49:0/2624336034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9668005040 con 0x7f96780f67c0 2023-11-30T21:46:22.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.471+0000 7f96777fe700 1 --2- 172.21.15.49:0/2624336034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96780f67c0 0x7f9678106560 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f9668013510 tx=0x7f966800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:22.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.471+0000 7f965ffff700 1 -- 172.21.15.49:0/2624336034 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9668009320 con 0x7f96780f67c0 2023-11-30T21:46:22.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.471+0000 7f967e062700 1 -- 172.21.15.49:0/2624336034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9678106c50 con 0x7f96780f67c0 2023-11-30T21:46:22.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.471+0000 7f965ffff700 1 -- 172.21.15.49:0/2624336034 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9668013f00 con 0x7f96780f67c0 2023-11-30T21:46:22.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.471+0000 7f965ffff700 1 -- 172.21.15.49:0/2624336034 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9668014d90 con 0x7f96780f67c0 2023-11-30T21:46:22.473 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.471+0000 7f967e062700 1 -- 172.21.15.49:0/2624336034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96781070e0 con 0x7f96780f67c0 2023-11-30T21:46:22.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.472+0000 7f965ffff700 1 -- 172.21.15.49:0/2624336034 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96680095f0 con 0x7f96780f67c0 2023-11-30T21:46:22.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.472+0000 7f967e062700 1 -- 172.21.15.49:0/2624336034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9678045ad0 con 0x7f96780f67c0 2023-11-30T21:46:22.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.472+0000 7f965ffff700 1 --2- 172.21.15.49:0/2624336034 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f966002ff60 0x7f9660032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:22.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.472+0000 7f965ffff700 1 -- 172.21.15.49:0/2624336034 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9668040fe0 con 0x7f96780f67c0 2023-11-30T21:46:22.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.472+0000 7f9676ffd700 1 --2- 172.21.15.49:0/2624336034 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f966002ff60 0x7f9660032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:22.475 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.473+0000 7f9676ffd700 1 --2- 172.21.15.49:0/2624336034 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f966002ff60 0x7f9660032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f966c0096a0 tx=0x7f966c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:22.477 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.477+0000 7f965ffff700 1 -- 172.21.15.49:0/2624336034 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f966801d7f0 con 0x7f96780f67c0 2023-11-30T21:46:22.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.698+0000 7f967e062700 1 -- 172.21.15.49:0/2624336034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9678107ee0 con 0x7f96780f67c0 2023-11-30T21:46:22.699 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.699+0000 7f965ffff700 1 -- 172.21.15.49:0/2624336034 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9668010230 con 0x7f96780f67c0 2023-11-30T21:46:22.700 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:22.700 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:22.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.700+0000 7f967e062700 1 -- 172.21.15.49:0/2624336034 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f966002ff60 msgr2=0x7f9660032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:22.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.700+0000 7f967e062700 1 --2- 172.21.15.49:0/2624336034 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f966002ff60 0x7f9660032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f966c0096a0 tx=0x7f966c006b40).stop 2023-11-30T21:46:22.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.700+0000 7f967e062700 1 -- 172.21.15.49:0/2624336034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96780f67c0 msgr2=0x7f9678106560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:22.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.700+0000 7f967e062700 1 --2- 172.21.15.49:0/2624336034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96780f67c0 0x7f9678106560 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f9668013510 tx=0x7f966800b040).stop 2023-11-30T21:46:22.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.700+0000 7f967e062700 1 -- 172.21.15.49:0/2624336034 shutdown_connections 2023-11-30T21:46:22.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.700+0000 7f967e062700 1 --2- 172.21.15.49:0/2624336034 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f966002ff60 0x7f9660032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:22.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.700+0000 7f967e062700 1 --2- 172.21.15.49:0/2624336034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96780f67c0 0x7f9678106560 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:22.702 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.700+0000 7f967e062700 1 -- 172.21.15.49:0/2624336034 >> 172.21.15.49:0/2624336034 conn(0x7f96780f00f0 msgr2=0x7f96780f4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:22.702 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.700+0000 7f967e062700 1 -- 172.21.15.49:0/2624336034 shutdown_connections 2023-11-30T21:46:22.702 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:22.700+0000 7f967e062700 1 -- 172.21.15.49:0/2624336034 wait complete. 2023-11-30T21:46:22.702 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:23.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:23 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:21.477390+0000 mgr.smithi027.brrjcl (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:23.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:23 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:22.699020+0000 mon.smithi027 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.49:0/2624336034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:24.092 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:24.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:25.364 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.362+0000 7f45ec90a700 1 -- 172.21.15.49:0/3275608476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e80f5950 msgr2=0x7f45e80f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:25.364 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.362+0000 7f45ec90a700 1 --2- 172.21.15.49:0/3275608476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e80f5950 0x7f45e80f5d70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f45d8009fb0 tx=0x7f45d8009b30).stop 2023-11-30T21:46:25.364 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.362+0000 7f45ec90a700 1 -- 172.21.15.49:0/3275608476 shutdown_connections 2023-11-30T21:46:25.364 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.362+0000 7f45ec90a700 1 --2- 172.21.15.49:0/3275608476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e80f5950 0x7f45e80f5d70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:25.364 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.362+0000 7f45ec90a700 1 -- 172.21.15.49:0/3275608476 >> 172.21.15.49:0/3275608476 conn(0x7f45e80f0260 msgr2=0x7f45e80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:25.364 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.362+0000 7f45ec90a700 1 -- 172.21.15.49:0/3275608476 shutdown_connections 2023-11-30T21:46:25.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.362+0000 7f45ec90a700 1 -- 172.21.15.49:0/3275608476 wait complete. 2023-11-30T21:46:25.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.362+0000 7f45ec90a700 1 Processor -- start 2023-11-30T21:46:25.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.362+0000 7f45ec90a700 1 -- start start 2023-11-30T21:46:25.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.363+0000 7f45ec90a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e80f5950 0x7f45e8106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:25.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.363+0000 7f45ec90a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e8106c30 con 0x7f45e80f5950 2023-11-30T21:46:25.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.363+0000 7f45e659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e80f5950 0x7f45e8106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:25.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.363+0000 7f45e659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e80f5950 0x7f45e8106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:49768/0 (socket says 172.21.15.49:49768) 2023-11-30T21:46:25.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.363+0000 7f45e659c700 1 -- 172.21.15.49:0/1344208369 learned_addr learned my addr 172.21.15.49:0/1344208369 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:25.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.363+0000 7f45e659c700 1 -- 172.21.15.49:0/1344208369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d8005040 con 0x7f45e80f5950 2023-11-30T21:46:25.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.363+0000 7f45e659c700 1 --2- 172.21.15.49:0/1344208369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e80f5950 0x7f45e8106740 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f45d8013510 tx=0x7f45d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:25.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.363+0000 7f45deffd700 1 -- 172.21.15.49:0/1344208369 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45d8009320 con 0x7f45e80f5950 2023-11-30T21:46:25.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.363+0000 7f45ec90a700 1 -- 172.21.15.49:0/1344208369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e8106e30 con 0x7f45e80f5950 2023-11-30T21:46:25.365 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.363+0000 7f45deffd700 1 -- 172.21.15.49:0/1344208369 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45d8013f00 con 0x7f45e80f5950 2023-11-30T21:46:25.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.364+0000 7f45deffd700 1 -- 172.21.15.49:0/1344208369 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45d8014d90 con 0x7f45e80f5950 2023-11-30T21:46:25.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.364+0000 7f45ec90a700 1 -- 172.21.15.49:0/1344208369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e81072c0 con 0x7f45e80f5950 2023-11-30T21:46:25.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.364+0000 7f45deffd700 1 -- 172.21.15.49:0/1344208369 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45d80095f0 con 0x7f45e80f5950 2023-11-30T21:46:25.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.364+0000 7f45ec90a700 1 -- 172.21.15.49:0/1344208369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e8045ad0 con 0x7f45e80f5950 2023-11-30T21:46:25.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.364+0000 7f45deffd700 1 --2- 172.21.15.49:0/1344208369 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f45d402ff60 0x7f45d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:25.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.364+0000 7f45deffd700 1 -- 172.21.15.49:0/1344208369 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45d80410a0 con 0x7f45e80f5950 2023-11-30T21:46:25.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.365+0000 7f45e5d9b700 1 --2- 172.21.15.49:0/1344208369 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f45d402ff60 0x7f45d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:25.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.365+0000 7f45e5d9b700 1 --2- 172.21.15.49:0/1344208369 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f45d402ff60 0x7f45d4032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f45d00096a0 tx=0x7f45d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:25.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.369+0000 7f45deffd700 1 -- 172.21.15.49:0/1344208369 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45d801d7f0 con 0x7f45e80f5950 2023-11-30T21:46:25.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:23.477785+0000 mgr.smithi027.brrjcl (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:25.589 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.589+0000 7f45ec90a700 1 -- 172.21.15.49:0/1344208369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45e8108170 con 0x7f45e80f5950 2023-11-30T21:46:25.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.589+0000 7f45deffd700 1 -- 172.21.15.49:0/1344208369 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f45d8010230 con 0x7f45e80f5950 2023-11-30T21:46:25.590 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:25.591 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.591+0000 7f45ec90a700 1 -- 172.21.15.49:0/1344208369 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f45d402ff60 msgr2=0x7f45d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.591+0000 7f45ec90a700 1 --2- 172.21.15.49:0/1344208369 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f45d402ff60 0x7f45d4032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f45d00096a0 tx=0x7f45d0006b40).stop 2023-11-30T21:46:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.591+0000 7f45ec90a700 1 -- 172.21.15.49:0/1344208369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e80f5950 msgr2=0x7f45e8106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.591+0000 7f45ec90a700 1 --2- 172.21.15.49:0/1344208369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e80f5950 0x7f45e8106740 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f45d8013510 tx=0x7f45d800b040).stop 2023-11-30T21:46:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.591+0000 7f45ec90a700 1 -- 172.21.15.49:0/1344208369 shutdown_connections 2023-11-30T21:46:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.591+0000 7f45ec90a700 1 --2- 172.21.15.49:0/1344208369 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f45d402ff60 0x7f45d4032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.591+0000 7f45ec90a700 1 --2- 172.21.15.49:0/1344208369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e80f5950 0x7f45e8106740 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.591+0000 7f45ec90a700 1 -- 172.21.15.49:0/1344208369 >> 172.21.15.49:0/1344208369 conn(0x7f45e80f0260 msgr2=0x7f45e80f4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:25.593 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.591+0000 7f45ec90a700 1 -- 172.21.15.49:0/1344208369 shutdown_connections 2023-11-30T21:46:25.593 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:25.591+0000 7f45ec90a700 1 -- 172.21.15.49:0/1344208369 wait complete. 2023-11-30T21:46:25.593 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:26.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:26 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:25.590026+0000 mon.smithi027 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.49:0/1344208369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:26.999 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:27.000 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:27.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:27 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:25.478134+0000 mgr.smithi027.brrjcl (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:28.198 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.195+0000 7efdb3a18700 1 -- 172.21.15.49:0/3712952917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac0f5410 msgr2=0x7efdac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:28.199 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.195+0000 7efdb3a18700 1 --2- 172.21.15.49:0/3712952917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac0f5410 0x7efdac0f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7efda8009fb0 tx=0x7efda8009b30).stop 2023-11-30T21:46:28.199 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.196+0000 7efdb3a18700 1 -- 172.21.15.49:0/3712952917 shutdown_connections 2023-11-30T21:46:28.199 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.196+0000 7efdb3a18700 1 --2- 172.21.15.49:0/3712952917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac0f5410 0x7efdac0f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:28.199 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.196+0000 7efdb3a18700 1 -- 172.21.15.49:0/3712952917 >> 172.21.15.49:0/3712952917 conn(0x7efdac0f0b70 msgr2=0x7efdac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:28.199 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.196+0000 7efdb3a18700 1 -- 172.21.15.49:0/3712952917 shutdown_connections 2023-11-30T21:46:28.199 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.196+0000 7efdb3a18700 1 -- 172.21.15.49:0/3712952917 wait complete. 2023-11-30T21:46:28.199 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.196+0000 7efdb3a18700 1 Processor -- start 2023-11-30T21:46:28.199 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.196+0000 7efdb3a18700 1 -- start start 2023-11-30T21:46:28.199 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.196+0000 7efdb3a18700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac0f5410 0x7efdac106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:28.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.196+0000 7efdb3a18700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdac106c10 con 0x7efdac0f5410 2023-11-30T21:46:28.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.197+0000 7efdb17b4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac0f5410 0x7efdac106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:28.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.197+0000 7efdb17b4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac0f5410 0x7efdac106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:49786/0 (socket says 172.21.15.49:49786) 2023-11-30T21:46:28.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.197+0000 7efdb17b4700 1 -- 172.21.15.49:0/1941052207 learned_addr learned my addr 172.21.15.49:0/1941052207 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:28.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.197+0000 7efdb17b4700 1 -- 172.21.15.49:0/1941052207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda8005040 con 0x7efdac0f5410 2023-11-30T21:46:28.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.197+0000 7efdb17b4700 1 --2- 172.21.15.49:0/1941052207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac0f5410 0x7efdac106720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7efda8013510 tx=0x7efda800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:28.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.197+0000 7efd9dffb700 1 -- 172.21.15.49:0/1941052207 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efda8009320 con 0x7efdac0f5410 2023-11-30T21:46:28.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.197+0000 7efd9dffb700 1 -- 172.21.15.49:0/1941052207 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efda8013f00 con 0x7efdac0f5410 2023-11-30T21:46:28.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.197+0000 7efdb3a18700 1 -- 172.21.15.49:0/1941052207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdac106e10 con 0x7efdac0f5410 2023-11-30T21:46:28.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.197+0000 7efd9dffb700 1 -- 172.21.15.49:0/1941052207 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efda8014d90 con 0x7efdac0f5410 2023-11-30T21:46:28.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.198+0000 7efdb3a18700 1 -- 172.21.15.49:0/1941052207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdac1072a0 con 0x7efdac0f5410 2023-11-30T21:46:28.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.198+0000 7efd9dffb700 1 -- 172.21.15.49:0/1941052207 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efda80095f0 con 0x7efdac0f5410 2023-11-30T21:46:28.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.198+0000 7efdb3a18700 1 -- 172.21.15.49:0/1941052207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdac045ad0 con 0x7efdac0f5410 2023-11-30T21:46:28.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.199+0000 7efd9dffb700 1 --2- 172.21.15.49:0/1941052207 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efd9802ff60 0x7efd98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:28.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.199+0000 7efd9dffb700 1 -- 172.21.15.49:0/1941052207 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efda8040fc0 con 0x7efdac0f5410 2023-11-30T21:46:28.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.199+0000 7efdb0fb3700 1 --2- 172.21.15.49:0/1941052207 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efd9802ff60 0x7efd98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:28.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.199+0000 7efdb0fb3700 1 --2- 172.21.15.49:0/1941052207 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efd9802ff60 0x7efd98032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7efda00096a0 tx=0x7efda0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:28.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.203+0000 7efd9dffb700 1 -- 172.21.15.49:0/1941052207 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efda8014420 con 0x7efdac0f5410 2023-11-30T21:46:28.426 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.426+0000 7efdb3a18700 1 -- 172.21.15.49:0/1941052207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efdac108150 con 0x7efdac0f5410 2023-11-30T21:46:28.427 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.427+0000 7efd9dffb700 1 -- 172.21.15.49:0/1941052207 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efda8010230 con 0x7efdac0f5410 2023-11-30T21:46:28.427 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:28.427 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:28.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.428+0000 7efdb3a18700 1 -- 172.21.15.49:0/1941052207 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efd9802ff60 msgr2=0x7efd98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:28.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.428+0000 7efdb3a18700 1 --2- 172.21.15.49:0/1941052207 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efd9802ff60 0x7efd98032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7efda00096a0 tx=0x7efda0006b40).stop 2023-11-30T21:46:28.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.428+0000 7efdb3a18700 1 -- 172.21.15.49:0/1941052207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac0f5410 msgr2=0x7efdac106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:28.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.428+0000 7efdb3a18700 1 --2- 172.21.15.49:0/1941052207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac0f5410 0x7efdac106720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7efda8013510 tx=0x7efda800b040).stop 2023-11-30T21:46:28.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.428+0000 7efdb3a18700 1 -- 172.21.15.49:0/1941052207 shutdown_connections 2023-11-30T21:46:28.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.428+0000 7efdb3a18700 1 --2- 172.21.15.49:0/1941052207 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efd9802ff60 0x7efd98032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:28.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.428+0000 7efdb3a18700 1 --2- 172.21.15.49:0/1941052207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdac0f5410 0x7efdac106720 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:28.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.428+0000 7efdb3a18700 1 -- 172.21.15.49:0/1941052207 >> 172.21.15.49:0/1941052207 conn(0x7efdac0f0b70 msgr2=0x7efdac05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:28.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.428+0000 7efdb3a18700 1 -- 172.21.15.49:0/1941052207 shutdown_connections 2023-11-30T21:46:28.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:28.429+0000 7efdb3a18700 1 -- 172.21.15.49:0/1941052207 wait complete. 2023-11-30T21:46:28.430 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:29.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:27.478465+0000 mgr.smithi027.brrjcl (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:29.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:28.427120+0000 mon.smithi027 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.49:0/1941052207' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:29.813 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:29.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:30.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.985+0000 7f6352920700 1 -- 172.21.15.49:0/1334965630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f634c0f5410 msgr2=0x7f634c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:30.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.985+0000 7f6352920700 1 --2- 172.21.15.49:0/1334965630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f634c0f5410 0x7f634c0f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f633c009fb0 tx=0x7f633c009b30).stop 2023-11-30T21:46:30.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.985+0000 7f6352920700 1 -- 172.21.15.49:0/1334965630 shutdown_connections 2023-11-30T21:46:30.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.985+0000 7f6352920700 1 --2- 172.21.15.49:0/1334965630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f634c0f5410 0x7f634c0f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:30.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.985+0000 7f6352920700 1 -- 172.21.15.49:0/1334965630 >> 172.21.15.49:0/1334965630 conn(0x7f634c0f0b70 msgr2=0x7f634c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:30.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.985+0000 7f6352920700 1 -- 172.21.15.49:0/1334965630 shutdown_connections 2023-11-30T21:46:30.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.985+0000 7f6352920700 1 -- 172.21.15.49:0/1334965630 wait complete. 2023-11-30T21:46:30.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.986+0000 7f6352920700 1 Processor -- start 2023-11-30T21:46:30.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.986+0000 7f6352920700 1 -- start start 2023-11-30T21:46:30.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.986+0000 7f6352920700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f634c0f5410 0x7f634c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:30.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.986+0000 7f6352920700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c105b20 con 0x7f634c0f5410 2023-11-30T21:46:30.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.987+0000 7f634bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f634c0f5410 0x7f634c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:30.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.987+0000 7f634bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f634c0f5410 0x7f634c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:45366/0 (socket says 172.21.15.49:45366) 2023-11-30T21:46:30.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.987+0000 7f634bfff700 1 -- 172.21.15.49:0/852488742 learned_addr learned my addr 172.21.15.49:0/852488742 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:30.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.987+0000 7f634bfff700 1 -- 172.21.15.49:0/852488742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c005040 con 0x7f634c0f5410 2023-11-30T21:46:30.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.987+0000 7f634bfff700 1 --2- 172.21.15.49:0/852488742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f634c0f5410 0x7f634c105630 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f633c013510 tx=0x7f633c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:30.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.987+0000 7f6348ff9700 1 -- 172.21.15.49:0/852488742 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f633c009320 con 0x7f634c0f5410 2023-11-30T21:46:30.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.987+0000 7f6352920700 1 -- 172.21.15.49:0/852488742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f634c105d20 con 0x7f634c0f5410 2023-11-30T21:46:30.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.987+0000 7f6348ff9700 1 -- 172.21.15.49:0/852488742 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f633c013f00 con 0x7f634c0f5410 2023-11-30T21:46:30.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.987+0000 7f6348ff9700 1 -- 172.21.15.49:0/852488742 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f633c014d90 con 0x7f634c0f5410 2023-11-30T21:46:30.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.987+0000 7f6352920700 1 -- 172.21.15.49:0/852488742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f634c1061b0 con 0x7f634c0f5410 2023-11-30T21:46:30.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.988+0000 7f6348ff9700 1 -- 172.21.15.49:0/852488742 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f633c0263a0 con 0x7f634c0f5410 2023-11-30T21:46:30.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.988+0000 7f6348ff9700 1 --2- 172.21.15.49:0/852488742 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f633402ffb0 0x7f6334032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:30.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.988+0000 7f6348ff9700 1 -- 172.21.15.49:0/852488742 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f633c041090 con 0x7f634c0f5410 2023-11-30T21:46:30.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.988+0000 7f6352920700 1 -- 172.21.15.49:0/852488742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f634c045ad0 con 0x7f634c0f5410 2023-11-30T21:46:30.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.989+0000 7f634b7fe700 1 --2- 172.21.15.49:0/852488742 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f633402ffb0 0x7f6334032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:30.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.989+0000 7f634b7fe700 1 --2- 172.21.15.49:0/852488742 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f633402ffb0 0x7f6334032470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f63400096a0 tx=0x7f6340006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:30.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:30.994+0000 7f6348ff9700 1 -- 172.21.15.49:0/852488742 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f633c012920 con 0x7f634c0f5410 2023-11-30T21:46:31.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:31.216+0000 7f6352920700 1 -- 172.21.15.49:0/852488742 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f634c0faf50 con 0x7f634c0f5410 2023-11-30T21:46:31.218 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:31.216+0000 7f6348ff9700 1 -- 172.21.15.49:0/852488742 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f633c02b020 con 0x7f634c0f5410 2023-11-30T21:46:31.218 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:31.218 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:31.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:31.218+0000 7f6352920700 1 -- 172.21.15.49:0/852488742 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f633402ffb0 msgr2=0x7f6334032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:31.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:31.218+0000 7f6352920700 1 --2- 172.21.15.49:0/852488742 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f633402ffb0 0x7f6334032470 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f63400096a0 tx=0x7f6340006b40).stop 2023-11-30T21:46:31.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:31.218+0000 7f6352920700 1 -- 172.21.15.49:0/852488742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f634c0f5410 msgr2=0x7f634c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:31.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:31.218+0000 7f6352920700 1 --2- 172.21.15.49:0/852488742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f634c0f5410 0x7f634c105630 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f633c013510 tx=0x7f633c00b040).stop 2023-11-30T21:46:31.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:31.218+0000 7f6352920700 1 -- 172.21.15.49:0/852488742 shutdown_connections 2023-11-30T21:46:31.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:31.218+0000 7f6352920700 1 --2- 172.21.15.49:0/852488742 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f633402ffb0 0x7f6334032470 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:31.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:31.218+0000 7f6352920700 1 --2- 172.21.15.49:0/852488742 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f634c0f5410 0x7f634c105630 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:31.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:31.218+0000 7f6352920700 1 -- 172.21.15.49:0/852488742 >> 172.21.15.49:0/852488742 conn(0x7f634c0f0b70 msgr2=0x7f634c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:31.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:31.218+0000 7f6352920700 1 -- 172.21.15.49:0/852488742 shutdown_connections 2023-11-30T21:46:31.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:31.218+0000 7f6352920700 1 -- 172.21.15.49:0/852488742 wait complete. 2023-11-30T21:46:31.220 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:31.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:29.478816+0000 mgr.smithi027.brrjcl (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:32.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:32 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:31.217945+0000 mon.smithi027 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.49:0/852488742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:32.601 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:32.601 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:33.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:33 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:31.479196+0000 mgr.smithi027.brrjcl (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:33.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.810+0000 7fdefe258700 1 -- 172.21.15.49:0/2357922949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef80f6510 msgr2=0x7fdef80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:33.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.810+0000 7fdefe258700 1 --2- 172.21.15.49:0/2357922949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef80f6510 0x7fdef80f6930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fdee8009fb0 tx=0x7fdee8009b30).stop 2023-11-30T21:46:33.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.811+0000 7fdefe258700 1 -- 172.21.15.49:0/2357922949 shutdown_connections 2023-11-30T21:46:33.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.811+0000 7fdefe258700 1 --2- 172.21.15.49:0/2357922949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef80f6510 0x7fdef80f6930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:33.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.811+0000 7fdefe258700 1 -- 172.21.15.49:0/2357922949 >> 172.21.15.49:0/2357922949 conn(0x7fdef80f1c70 msgr2=0x7fdef80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:33.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.811+0000 7fdefe258700 1 -- 172.21.15.49:0/2357922949 shutdown_connections 2023-11-30T21:46:33.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.811+0000 7fdefe258700 1 -- 172.21.15.49:0/2357922949 wait complete. 2023-11-30T21:46:33.818 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.811+0000 7fdefe258700 1 Processor -- start 2023-11-30T21:46:33.818 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.812+0000 7fdefe258700 1 -- start start 2023-11-30T21:46:33.818 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.812+0000 7fdefe258700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef80f6510 0x7fdef8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:33.818 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.812+0000 7fdefe258700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef8106c20 con 0x7fdef80f6510 2023-11-30T21:46:33.818 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.812+0000 7fdef77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef80f6510 0x7fdef8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:33.819 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.812+0000 7fdef77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef80f6510 0x7fdef8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:45388/0 (socket says 172.21.15.49:45388) 2023-11-30T21:46:33.819 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.812+0000 7fdef77fe700 1 -- 172.21.15.49:0/3388678992 learned_addr learned my addr 172.21.15.49:0/3388678992 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:33.819 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.813+0000 7fdef77fe700 1 -- 172.21.15.49:0/3388678992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee8005040 con 0x7fdef80f6510 2023-11-30T21:46:33.819 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.813+0000 7fdef77fe700 1 --2- 172.21.15.49:0/3388678992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef80f6510 0x7fdef8106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fdee8013510 tx=0x7fdee800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:33.819 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.813+0000 7fdeeffff700 1 -- 172.21.15.49:0/3388678992 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdee8009320 con 0x7fdef80f6510 2023-11-30T21:46:33.819 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.813+0000 7fdeeffff700 1 -- 172.21.15.49:0/3388678992 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdee8013f00 con 0x7fdef80f6510 2023-11-30T21:46:33.819 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.813+0000 7fdeeffff700 1 -- 172.21.15.49:0/3388678992 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdee8014d90 con 0x7fdef80f6510 2023-11-30T21:46:33.819 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.813+0000 7fdefe258700 1 -- 172.21.15.49:0/3388678992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef8106e20 con 0x7fdef80f6510 2023-11-30T21:46:33.819 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.813+0000 7fdefe258700 1 -- 172.21.15.49:0/3388678992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef81072b0 con 0x7fdef80f6510 2023-11-30T21:46:33.819 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.814+0000 7fdeeffff700 1 -- 172.21.15.49:0/3388678992 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdee8014420 con 0x7fdef80f6510 2023-11-30T21:46:33.819 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.814+0000 7fdefe258700 1 -- 172.21.15.49:0/3388678992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef8045ad0 con 0x7fdef80f6510 2023-11-30T21:46:33.819 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.814+0000 7fdeeffff700 1 --2- 172.21.15.49:0/3388678992 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdee402ff60 0x7fdee4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:33.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.814+0000 7fdeeffff700 1 -- 172.21.15.49:0/3388678992 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdee80410c0 con 0x7fdef80f6510 2023-11-30T21:46:33.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.814+0000 7fdef6ffd700 1 --2- 172.21.15.49:0/3388678992 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdee402ff60 0x7fdee4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:33.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.815+0000 7fdef6ffd700 1 --2- 172.21.15.49:0/3388678992 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdee402ff60 0x7fdee4032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fdee00096a0 tx=0x7fdee0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:33.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:33.819+0000 7fdeeffff700 1 -- 172.21.15.49:0/3388678992 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdee80095f0 con 0x7fdef80f6510 2023-11-30T21:46:34.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:34.038+0000 7fdefe258700 1 -- 172.21.15.49:0/3388678992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdef80fbf60 con 0x7fdef80f6510 2023-11-30T21:46:34.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:34.039+0000 7fdeeffff700 1 -- 172.21.15.49:0/3388678992 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdee801db70 con 0x7fdef80f6510 2023-11-30T21:46:34.040 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:34.041 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:34.042 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:34.040+0000 7fdefe258700 1 -- 172.21.15.49:0/3388678992 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdee402ff60 msgr2=0x7fdee4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:34.042 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:34.040+0000 7fdefe258700 1 --2- 172.21.15.49:0/3388678992 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdee402ff60 0x7fdee4032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fdee00096a0 tx=0x7fdee0006b40).stop 2023-11-30T21:46:34.042 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:34.040+0000 7fdefe258700 1 -- 172.21.15.49:0/3388678992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef80f6510 msgr2=0x7fdef8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:34.042 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:34.040+0000 7fdefe258700 1 --2- 172.21.15.49:0/3388678992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef80f6510 0x7fdef8106730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fdee8013510 tx=0x7fdee800b040).stop 2023-11-30T21:46:34.043 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:34.040+0000 7fdefe258700 1 -- 172.21.15.49:0/3388678992 shutdown_connections 2023-11-30T21:46:34.043 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:34.040+0000 7fdefe258700 1 --2- 172.21.15.49:0/3388678992 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdee402ff60 0x7fdee4032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:34.043 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:34.040+0000 7fdefe258700 1 --2- 172.21.15.49:0/3388678992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdef80f6510 0x7fdef8106730 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:34.043 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:34.040+0000 7fdefe258700 1 -- 172.21.15.49:0/3388678992 >> 172.21.15.49:0/3388678992 conn(0x7fdef80f1c70 msgr2=0x7fdef80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:34.043 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:34.041+0000 7fdefe258700 1 -- 172.21.15.49:0/3388678992 shutdown_connections 2023-11-30T21:46:34.043 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:34.041+0000 7fdefe258700 1 -- 172.21.15.49:0/3388678992 wait complete. 2023-11-30T21:46:34.043 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:34.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:34 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:34.040342+0000 mon.smithi027 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.49:0/3388678992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:35.457 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:35.458 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:35.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:35 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:33.479514+0000 mgr.smithi027.brrjcl (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:36.659 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.657+0000 7f7c59d41700 1 -- 172.21.15.49:0/1408866211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c540f51a0 msgr2=0x7f7c540f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:36.659 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.657+0000 7f7c59d41700 1 --2- 172.21.15.49:0/1408866211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c540f51a0 0x7f7c540f55c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f7c48009fb0 tx=0x7f7c48009b30).stop 2023-11-30T21:46:36.659 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.657+0000 7f7c59d41700 1 -- 172.21.15.49:0/1408866211 shutdown_connections 2023-11-30T21:46:36.659 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.657+0000 7f7c59d41700 1 --2- 172.21.15.49:0/1408866211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c540f51a0 0x7f7c540f55c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:36.659 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.657+0000 7f7c59d41700 1 -- 172.21.15.49:0/1408866211 >> 172.21.15.49:0/1408866211 conn(0x7f7c540f09a0 msgr2=0x7f7c540f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:36.659 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.657+0000 7f7c59d41700 1 -- 172.21.15.49:0/1408866211 shutdown_connections 2023-11-30T21:46:36.660 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.657+0000 7f7c59d41700 1 -- 172.21.15.49:0/1408866211 wait complete. 2023-11-30T21:46:36.660 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.657+0000 7f7c59d41700 1 Processor -- start 2023-11-30T21:46:36.660 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.657+0000 7f7c59d41700 1 -- start start 2023-11-30T21:46:36.660 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.658+0000 7f7c59d41700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c540f51a0 0x7f7c54109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:36.660 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.658+0000 7f7c59d41700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c54060280 con 0x7f7c540f51a0 2023-11-30T21:46:36.663 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.658+0000 7f7c58d3f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c540f51a0 0x7f7c54109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:36.663 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.658+0000 7f7c58d3f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c540f51a0 0x7f7c54109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:45408/0 (socket says 172.21.15.49:45408) 2023-11-30T21:46:36.663 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.658+0000 7f7c58d3f700 1 -- 172.21.15.49:0/1677485424 learned_addr learned my addr 172.21.15.49:0/1677485424 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:36.663 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.658+0000 7f7c58d3f700 1 -- 172.21.15.49:0/1677485424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c48005040 con 0x7f7c540f51a0 2023-11-30T21:46:36.664 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.658+0000 7f7c58d3f700 1 --2- 172.21.15.49:0/1677485424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c540f51a0 0x7f7c54109170 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f7c48013510 tx=0x7f7c4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:36.664 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.659+0000 7f7c517fa700 1 -- 172.21.15.49:0/1677485424 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c48009320 con 0x7f7c540f51a0 2023-11-30T21:46:36.664 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.659+0000 7f7c59d41700 1 -- 172.21.15.49:0/1677485424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c54060480 con 0x7f7c540f51a0 2023-11-30T21:46:36.664 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.659+0000 7f7c517fa700 1 -- 172.21.15.49:0/1677485424 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c48013f00 con 0x7f7c540f51a0 2023-11-30T21:46:36.664 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.659+0000 7f7c517fa700 1 -- 172.21.15.49:0/1677485424 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c48014dc0 con 0x7f7c540f51a0 2023-11-30T21:46:36.664 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.659+0000 7f7c59d41700 1 -- 172.21.15.49:0/1677485424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c541098a0 con 0x7f7c540f51a0 2023-11-30T21:46:36.664 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.660+0000 7f7c517fa700 1 -- 172.21.15.49:0/1677485424 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c48014420 con 0x7f7c540f51a0 2023-11-30T21:46:36.664 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.660+0000 7f7c59d41700 1 -- 172.21.15.49:0/1677485424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c54045ad0 con 0x7f7c540f51a0 2023-11-30T21:46:36.664 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.660+0000 7f7c517fa700 1 --2- 172.21.15.49:0/1677485424 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7c4402ff60 0x7f7c44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:36.664 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.660+0000 7f7c517fa700 1 -- 172.21.15.49:0/1677485424 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c48041050 con 0x7f7c540f51a0 2023-11-30T21:46:36.664 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.660+0000 7f7c53fff700 1 --2- 172.21.15.49:0/1677485424 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7c4402ff60 0x7f7c44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:36.665 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.660+0000 7f7c53fff700 1 --2- 172.21.15.49:0/1677485424 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7c4402ff60 0x7f7c44032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f7c400096a0 tx=0x7f7c40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:36.666 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.665+0000 7f7c517fa700 1 -- 172.21.15.49:0/1677485424 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c480095f0 con 0x7f7c540f51a0 2023-11-30T21:46:36.886 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.885+0000 7f7c59d41700 1 -- 172.21.15.49:0/1677485424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c540fabe0 con 0x7f7c540f51a0 2023-11-30T21:46:36.887 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.885+0000 7f7c517fa700 1 -- 172.21.15.49:0/1677485424 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7c4801cb10 con 0x7f7c540f51a0 2023-11-30T21:46:36.887 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:36.887 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:36.890 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.887+0000 7f7c59d41700 1 -- 172.21.15.49:0/1677485424 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7c4402ff60 msgr2=0x7f7c44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:36.890 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.887+0000 7f7c59d41700 1 --2- 172.21.15.49:0/1677485424 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7c4402ff60 0x7f7c44032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f7c400096a0 tx=0x7f7c40006b40).stop 2023-11-30T21:46:36.890 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.887+0000 7f7c59d41700 1 -- 172.21.15.49:0/1677485424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c540f51a0 msgr2=0x7f7c54109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:36.890 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.887+0000 7f7c59d41700 1 --2- 172.21.15.49:0/1677485424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c540f51a0 0x7f7c54109170 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f7c48013510 tx=0x7f7c4800b040).stop 2023-11-30T21:46:36.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.887+0000 7f7c59d41700 1 -- 172.21.15.49:0/1677485424 shutdown_connections 2023-11-30T21:46:36.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.887+0000 7f7c59d41700 1 --2- 172.21.15.49:0/1677485424 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7c4402ff60 0x7f7c44032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:36.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.887+0000 7f7c59d41700 1 --2- 172.21.15.49:0/1677485424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c540f51a0 0x7f7c54109170 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:36.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.887+0000 7f7c59d41700 1 -- 172.21.15.49:0/1677485424 >> 172.21.15.49:0/1677485424 conn(0x7f7c540f09a0 msgr2=0x7f7c540f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:36.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.887+0000 7f7c59d41700 1 -- 172.21.15.49:0/1677485424 shutdown_connections 2023-11-30T21:46:36.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:36.887+0000 7f7c59d41700 1 -- 172.21.15.49:0/1677485424 wait complete. 2023-11-30T21:46:36.891 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:37.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:37 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:35.479799+0000 mgr.smithi027.brrjcl (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:37.516 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:37 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:36.886857+0000 mon.smithi027 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.49:0/1677485424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:38.291 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:38.291 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:39.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.492+0000 7fdff0c55700 1 -- 172.21.15.49:0/1170365317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec060220 msgr2=0x7fdfec0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:39.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.492+0000 7fdff0c55700 1 --2- 172.21.15.49:0/1170365317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec060220 0x7fdfec0f9150 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fdfdc009fb0 tx=0x7fdfdc009b30).stop 2023-11-30T21:46:39.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.492+0000 7fdff0c55700 1 -- 172.21.15.49:0/1170365317 shutdown_connections 2023-11-30T21:46:39.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.492+0000 7fdff0c55700 1 --2- 172.21.15.49:0/1170365317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec060220 0x7fdfec0f9150 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:39.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.492+0000 7fdff0c55700 1 -- 172.21.15.49:0/1170365317 >> 172.21.15.49:0/1170365317 conn(0x7fdfec0f0b70 msgr2=0x7fdfec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:39.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.492+0000 7fdff0c55700 1 -- 172.21.15.49:0/1170365317 shutdown_connections 2023-11-30T21:46:39.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.492+0000 7fdff0c55700 1 -- 172.21.15.49:0/1170365317 wait complete. 2023-11-30T21:46:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.492+0000 7fdff0c55700 1 Processor -- start 2023-11-30T21:46:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.492+0000 7fdff0c55700 1 -- start start 2023-11-30T21:46:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.493+0000 7fdff0c55700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec060220 0x7fdfec1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.493+0000 7fdff0c55700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec106bd0 con 0x7fdfec060220 2023-11-30T21:46:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.493+0000 7fdfea59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec060220 0x7fdfec1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.493+0000 7fdfea59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec060220 0x7fdfec1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:35758/0 (socket says 172.21.15.49:35758) 2023-11-30T21:46:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.493+0000 7fdfea59c700 1 -- 172.21.15.49:0/1664586751 learned_addr learned my addr 172.21.15.49:0/1664586751 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.493+0000 7fdfea59c700 1 -- 172.21.15.49:0/1664586751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc005040 con 0x7fdfec060220 2023-11-30T21:46:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.494+0000 7fdfea59c700 1 --2- 172.21.15.49:0/1664586751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec060220 0x7fdfec1066e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fdfdc013510 tx=0x7fdfdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.494+0000 7fdfdaffd700 1 -- 172.21.15.49:0/1664586751 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfdc009320 con 0x7fdfec060220 2023-11-30T21:46:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.494+0000 7fdff0c55700 1 -- 172.21.15.49:0/1664586751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfec106dd0 con 0x7fdfec060220 2023-11-30T21:46:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.494+0000 7fdfdaffd700 1 -- 172.21.15.49:0/1664586751 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfdc013f00 con 0x7fdfec060220 2023-11-30T21:46:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.494+0000 7fdfdaffd700 1 -- 172.21.15.49:0/1664586751 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfdc014dc0 con 0x7fdfec060220 2023-11-30T21:46:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.494+0000 7fdff0c55700 1 -- 172.21.15.49:0/1664586751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfec107260 con 0x7fdfec060220 2023-11-30T21:46:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.495+0000 7fdfdaffd700 1 -- 172.21.15.49:0/1664586751 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfdc01d3d0 con 0x7fdfec060220 2023-11-30T21:46:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.495+0000 7fdfdaffd700 1 --2- 172.21.15.49:0/1664586751 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdfd402ff60 0x7fdfd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.495+0000 7fdfdaffd700 1 -- 172.21.15.49:0/1664586751 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfdc041100 con 0x7fdfec060220 2023-11-30T21:46:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.495+0000 7fdff0c55700 1 -- 172.21.15.49:0/1664586751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfec045ad0 con 0x7fdfec060220 2023-11-30T21:46:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.495+0000 7fdfe9d9b700 1 --2- 172.21.15.49:0/1664586751 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdfd402ff60 0x7fdfd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.495+0000 7fdfe9d9b700 1 --2- 172.21.15.49:0/1664586751 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdfd402ff60 0x7fdfd4032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fdfe00096a0 tx=0x7fdfe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:39.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.500+0000 7fdfdaffd700 1 -- 172.21.15.49:0/1664586751 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfdc012920 con 0x7fdfec060220 2023-11-30T21:46:39.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:37.480108+0000 mgr.smithi027.brrjcl (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:39.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.720+0000 7fdff0c55700 1 -- 172.21.15.49:0/1664586751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfec108110 con 0x7fdfec060220 2023-11-30T21:46:39.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.721+0000 7fdfdaffd700 1 -- 172.21.15.49:0/1664586751 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdfdc02b020 con 0x7fdfec060220 2023-11-30T21:46:39.723 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:39.723 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:39.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.722+0000 7fdff0c55700 1 -- 172.21.15.49:0/1664586751 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdfd402ff60 msgr2=0x7fdfd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:39.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.722+0000 7fdff0c55700 1 --2- 172.21.15.49:0/1664586751 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdfd402ff60 0x7fdfd4032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fdfe00096a0 tx=0x7fdfe0006b40).stop 2023-11-30T21:46:39.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.722+0000 7fdff0c55700 1 -- 172.21.15.49:0/1664586751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec060220 msgr2=0x7fdfec1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:39.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.722+0000 7fdff0c55700 1 --2- 172.21.15.49:0/1664586751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec060220 0x7fdfec1066e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fdfdc013510 tx=0x7fdfdc00b040).stop 2023-11-30T21:46:39.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.722+0000 7fdff0c55700 1 -- 172.21.15.49:0/1664586751 shutdown_connections 2023-11-30T21:46:39.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.722+0000 7fdff0c55700 1 --2- 172.21.15.49:0/1664586751 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdfd402ff60 0x7fdfd4032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:39.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.722+0000 7fdff0c55700 1 --2- 172.21.15.49:0/1664586751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec060220 0x7fdfec1066e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:39.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.722+0000 7fdff0c55700 1 -- 172.21.15.49:0/1664586751 >> 172.21.15.49:0/1664586751 conn(0x7fdfec0f0b70 msgr2=0x7fdfec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:39.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.722+0000 7fdff0c55700 1 -- 172.21.15.49:0/1664586751 shutdown_connections 2023-11-30T21:46:39.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:39.722+0000 7fdff0c55700 1 -- 172.21.15.49:0/1664586751 wait complete. 2023-11-30T21:46:39.725 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:40.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:40 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:39.722279+0000 mon.smithi027 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.49:0/1664586751' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:41.145 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:41.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:41.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:41 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:39.480454+0000 mgr.smithi027.brrjcl (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:42.301 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.298+0000 7f519dd3d700 1 -- 172.21.15.49:0/3142834452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51980f62a0 msgr2=0x7f51980f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:42.301 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.298+0000 7f519dd3d700 1 --2- 172.21.15.49:0/3142834452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51980f62a0 0x7f51980f66c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f518c009fb0 tx=0x7f518c009b30).stop 2023-11-30T21:46:42.301 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.298+0000 7f519dd3d700 1 -- 172.21.15.49:0/3142834452 shutdown_connections 2023-11-30T21:46:42.301 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.298+0000 7f519dd3d700 1 --2- 172.21.15.49:0/3142834452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51980f62a0 0x7f51980f66c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:42.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.298+0000 7f519dd3d700 1 -- 172.21.15.49:0/3142834452 >> 172.21.15.49:0/3142834452 conn(0x7f51980f1aa0 msgr2=0x7f51980f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:42.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.298+0000 7f519dd3d700 1 -- 172.21.15.49:0/3142834452 shutdown_connections 2023-11-30T21:46:42.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.298+0000 7f519dd3d700 1 -- 172.21.15.49:0/3142834452 wait complete. 2023-11-30T21:46:42.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.298+0000 7f519dd3d700 1 Processor -- start 2023-11-30T21:46:42.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.298+0000 7f519dd3d700 1 -- start start 2023-11-30T21:46:42.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.299+0000 7f519dd3d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51980f62a0 0x7f51981064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:42.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.299+0000 7f519dd3d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51981069a0 con 0x7f51980f62a0 2023-11-30T21:46:42.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.299+0000 7f519cd3b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51980f62a0 0x7f51981064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:42.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.299+0000 7f519cd3b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51980f62a0 0x7f51981064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:35774/0 (socket says 172.21.15.49:35774) 2023-11-30T21:46:42.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.299+0000 7f519cd3b700 1 -- 172.21.15.49:0/853521989 learned_addr learned my addr 172.21.15.49:0/853521989 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:42.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.299+0000 7f519cd3b700 1 -- 172.21.15.49:0/853521989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c005040 con 0x7f51980f62a0 2023-11-30T21:46:42.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.300+0000 7f519cd3b700 1 --2- 172.21.15.49:0/853521989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51980f62a0 0x7f51981064b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f518c013510 tx=0x7f518c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:42.303 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.300+0000 7f51957fa700 1 -- 172.21.15.49:0/853521989 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f518c009320 con 0x7f51980f62a0 2023-11-30T21:46:42.303 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.300+0000 7f51957fa700 1 -- 172.21.15.49:0/853521989 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f518c013f00 con 0x7f51980f62a0 2023-11-30T21:46:42.303 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.300+0000 7f519dd3d700 1 -- 172.21.15.49:0/853521989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5198106ba0 con 0x7f51980f62a0 2023-11-30T21:46:42.303 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.300+0000 7f51957fa700 1 -- 172.21.15.49:0/853521989 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f518c014dc0 con 0x7f51980f62a0 2023-11-30T21:46:42.303 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.300+0000 7f519dd3d700 1 -- 172.21.15.49:0/853521989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5198107030 con 0x7f51980f62a0 2023-11-30T21:46:42.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.301+0000 7f51957fa700 1 -- 172.21.15.49:0/853521989 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f518c0095f0 con 0x7f51980f62a0 2023-11-30T21:46:42.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.301+0000 7f519dd3d700 1 -- 172.21.15.49:0/853521989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5198045ad0 con 0x7f51980f62a0 2023-11-30T21:46:42.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.301+0000 7f51957fa700 1 --2- 172.21.15.49:0/853521989 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f518802ff60 0x7f5188032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:42.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.301+0000 7f51957fa700 1 -- 172.21.15.49:0/853521989 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f518c041000 con 0x7f51980f62a0 2023-11-30T21:46:42.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.301+0000 7f5197fff700 1 --2- 172.21.15.49:0/853521989 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f518802ff60 0x7f5188032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:42.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.302+0000 7f5197fff700 1 --2- 172.21.15.49:0/853521989 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f518802ff60 0x7f5188032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f51840096a0 tx=0x7f5184006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:42.307 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.306+0000 7f51957fa700 1 -- 172.21.15.49:0/853521989 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f518c014420 con 0x7f51980f62a0 2023-11-30T21:46:42.527 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.525+0000 7f519dd3d700 1 -- 172.21.15.49:0/853521989 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5198107ee0 con 0x7f51980f62a0 2023-11-30T21:46:42.528 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.526+0000 7f51957fa700 1 -- 172.21.15.49:0/853521989 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f518c010230 con 0x7f51980f62a0 2023-11-30T21:46:42.528 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:42.528 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:42.529 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.528+0000 7f519dd3d700 1 -- 172.21.15.49:0/853521989 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f518802ff60 msgr2=0x7f5188032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:42.529 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.528+0000 7f519dd3d700 1 --2- 172.21.15.49:0/853521989 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f518802ff60 0x7f5188032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f51840096a0 tx=0x7f5184006b40).stop 2023-11-30T21:46:42.529 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.528+0000 7f519dd3d700 1 -- 172.21.15.49:0/853521989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51980f62a0 msgr2=0x7f51981064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:42.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.528+0000 7f519dd3d700 1 --2- 172.21.15.49:0/853521989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51980f62a0 0x7f51981064b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f518c013510 tx=0x7f518c00b040).stop 2023-11-30T21:46:42.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.528+0000 7f519dd3d700 1 -- 172.21.15.49:0/853521989 shutdown_connections 2023-11-30T21:46:42.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.528+0000 7f519dd3d700 1 --2- 172.21.15.49:0/853521989 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f518802ff60 0x7f5188032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:42.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.528+0000 7f519dd3d700 1 --2- 172.21.15.49:0/853521989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51980f62a0 0x7f51981064b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:42.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.528+0000 7f519dd3d700 1 -- 172.21.15.49:0/853521989 >> 172.21.15.49:0/853521989 conn(0x7f51980f1aa0 msgr2=0x7f51980f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:42.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.528+0000 7f519dd3d700 1 -- 172.21.15.49:0/853521989 shutdown_connections 2023-11-30T21:46:42.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:42.528+0000 7f519dd3d700 1 -- 172.21.15.49:0/853521989 wait complete. 2023-11-30T21:46:42.530 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:43.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:41.480826+0000 mgr.smithi027.brrjcl (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:43.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:42.527538+0000 mon.smithi027 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.49:0/853521989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:43.912 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:43.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:45.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.132+0000 7f71ae175700 1 -- 172.21.15.49:0/4074874364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a80f6510 msgr2=0x7f71a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:45.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.132+0000 7f71ae175700 1 --2- 172.21.15.49:0/4074874364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a80f6510 0x7f71a80f6930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f719c009fb0 tx=0x7f719c009b30).stop 2023-11-30T21:46:45.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.132+0000 7f71ae175700 1 -- 172.21.15.49:0/4074874364 shutdown_connections 2023-11-30T21:46:45.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.132+0000 7f71ae175700 1 --2- 172.21.15.49:0/4074874364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a80f6510 0x7f71a80f6930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:45.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.132+0000 7f71ae175700 1 -- 172.21.15.49:0/4074874364 >> 172.21.15.49:0/4074874364 conn(0x7f71a80f1c70 msgr2=0x7f71a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:45.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.132+0000 7f71ae175700 1 -- 172.21.15.49:0/4074874364 shutdown_connections 2023-11-30T21:46:45.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.132+0000 7f71ae175700 1 -- 172.21.15.49:0/4074874364 wait complete. 2023-11-30T21:46:45.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.133+0000 7f71ae175700 1 Processor -- start 2023-11-30T21:46:45.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.133+0000 7f71ae175700 1 -- start start 2023-11-30T21:46:45.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.133+0000 7f71ae175700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a80f6510 0x7f71a806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:45.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.133+0000 7f71ae175700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a8069e20 con 0x7f71a80f6510 2023-11-30T21:46:45.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.134+0000 7f71a77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a80f6510 0x7f71a806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:45.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.134+0000 7f71a77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a80f6510 0x7f71a806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:35782/0 (socket says 172.21.15.49:35782) 2023-11-30T21:46:45.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.134+0000 7f71a77fe700 1 -- 172.21.15.49:0/2439598542 learned_addr learned my addr 172.21.15.49:0/2439598542 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:45.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.134+0000 7f71a77fe700 1 -- 172.21.15.49:0/2439598542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c005040 con 0x7f71a80f6510 2023-11-30T21:46:45.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.134+0000 7f71a77fe700 1 --2- 172.21.15.49:0/2439598542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a80f6510 0x7f71a806b9a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f719c013510 tx=0x7f719c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:45.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.135+0000 7f718ffff700 1 -- 172.21.15.49:0/2439598542 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f719c009320 con 0x7f71a80f6510 2023-11-30T21:46:45.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.135+0000 7f71ae175700 1 -- 172.21.15.49:0/2439598542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a8069fc0 con 0x7f71a80f6510 2023-11-30T21:46:45.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.135+0000 7f71ae175700 1 -- 172.21.15.49:0/2439598542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a806a450 con 0x7f71a80f6510 2023-11-30T21:46:45.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.135+0000 7f718ffff700 1 -- 172.21.15.49:0/2439598542 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f719c013f00 con 0x7f71a80f6510 2023-11-30T21:46:45.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.135+0000 7f718ffff700 1 -- 172.21.15.49:0/2439598542 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f719c014d60 con 0x7f71a80f6510 2023-11-30T21:46:45.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.135+0000 7f718ffff700 1 -- 172.21.15.49:0/2439598542 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f719c014420 con 0x7f71a80f6510 2023-11-30T21:46:45.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.135+0000 7f71ae175700 1 -- 172.21.15.49:0/2439598542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a8059100 con 0x7f71a80f6510 2023-11-30T21:46:45.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.136+0000 7f718ffff700 1 --2- 172.21.15.49:0/2439598542 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f719002ff60 0x7f7190032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:45.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.136+0000 7f718ffff700 1 -- 172.21.15.49:0/2439598542 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f719c041320 con 0x7f71a80f6510 2023-11-30T21:46:45.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.136+0000 7f71a6ffd700 1 --2- 172.21.15.49:0/2439598542 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f719002ff60 0x7f7190032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:45.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.136+0000 7f71a6ffd700 1 --2- 172.21.15.49:0/2439598542 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f719002ff60 0x7f7190032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f71980096a0 tx=0x7f7198006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:45.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.141+0000 7f718ffff700 1 -- 172.21.15.49:0/2439598542 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f719c01c320 con 0x7f71a80f6510 2023-11-30T21:46:45.361 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.359+0000 7f71ae175700 1 -- 172.21.15.49:0/2439598542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71a80fbf60 con 0x7f71a80f6510 2023-11-30T21:46:45.361 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.360+0000 7f718ffff700 1 -- 172.21.15.49:0/2439598542 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f719c012b20 con 0x7f71a80f6510 2023-11-30T21:46:45.361 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:45.362 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:45.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.361+0000 7f71ae175700 1 -- 172.21.15.49:0/2439598542 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f719002ff60 msgr2=0x7f7190032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:45.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.361+0000 7f71ae175700 1 --2- 172.21.15.49:0/2439598542 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f719002ff60 0x7f7190032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f71980096a0 tx=0x7f7198006b40).stop 2023-11-30T21:46:45.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.361+0000 7f71ae175700 1 -- 172.21.15.49:0/2439598542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a80f6510 msgr2=0x7f71a806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:45.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.361+0000 7f71ae175700 1 --2- 172.21.15.49:0/2439598542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a80f6510 0x7f71a806b9a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f719c013510 tx=0x7f719c00b040).stop 2023-11-30T21:46:45.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.361+0000 7f71ae175700 1 -- 172.21.15.49:0/2439598542 shutdown_connections 2023-11-30T21:46:45.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.361+0000 7f71ae175700 1 --2- 172.21.15.49:0/2439598542 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f719002ff60 0x7f7190032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:45.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.361+0000 7f71ae175700 1 --2- 172.21.15.49:0/2439598542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a80f6510 0x7f71a806b9a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:45.364 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.361+0000 7f71ae175700 1 -- 172.21.15.49:0/2439598542 >> 172.21.15.49:0/2439598542 conn(0x7f71a80f1c70 msgr2=0x7f71a80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:45.364 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.362+0000 7f71ae175700 1 -- 172.21.15.49:0/2439598542 shutdown_connections 2023-11-30T21:46:45.364 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:45.362+0000 7f71ae175700 1 -- 172.21.15.49:0/2439598542 wait complete. 2023-11-30T21:46:45.364 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:45 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:43.481153+0000 mgr.smithi027.brrjcl (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:46.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:46 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:45.361452+0000 mon.smithi027 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.49:0/2439598542' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:46.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:46.747 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:47.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:47 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:45.481487+0000 mgr.smithi027.brrjcl (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:47.934 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.931+0000 7f2373938700 1 -- 172.21.15.49:0/3712494114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c0f6510 msgr2=0x7f236c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:47.934 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.931+0000 7f2373938700 1 --2- 172.21.15.49:0/3712494114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c0f6510 0x7f236c0f6930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f2368009fb0 tx=0x7f2368009b30).stop 2023-11-30T21:46:47.934 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.931+0000 7f2373938700 1 -- 172.21.15.49:0/3712494114 shutdown_connections 2023-11-30T21:46:47.934 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.931+0000 7f2373938700 1 --2- 172.21.15.49:0/3712494114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c0f6510 0x7f236c0f6930 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:47.934 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.931+0000 7f2373938700 1 -- 172.21.15.49:0/3712494114 >> 172.21.15.49:0/3712494114 conn(0x7f236c0f1c70 msgr2=0x7f236c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:47.934 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.931+0000 7f2373938700 1 -- 172.21.15.49:0/3712494114 shutdown_connections 2023-11-30T21:46:47.934 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.931+0000 7f2373938700 1 -- 172.21.15.49:0/3712494114 wait complete. 2023-11-30T21:46:47.934 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.931+0000 7f2373938700 1 Processor -- start 2023-11-30T21:46:47.934 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.932+0000 7f2373938700 1 -- start start 2023-11-30T21:46:47.934 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.932+0000 7f2373938700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c0f6510 0x7f236c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:47.934 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.932+0000 7f2373938700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c106c20 con 0x7f236c0f6510 2023-11-30T21:46:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.932+0000 7f23716d4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c0f6510 0x7f236c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.932+0000 7f23716d4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c0f6510 0x7f236c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:35792/0 (socket says 172.21.15.49:35792) 2023-11-30T21:46:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.932+0000 7f23716d4700 1 -- 172.21.15.49:0/3912832424 learned_addr learned my addr 172.21.15.49:0/3912832424 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.932+0000 7f23716d4700 1 -- 172.21.15.49:0/3912832424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2368005040 con 0x7f236c0f6510 2023-11-30T21:46:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.932+0000 7f23716d4700 1 --2- 172.21.15.49:0/3912832424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c0f6510 0x7f236c106730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f2368013510 tx=0x7f236800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.933+0000 7f2361ffb700 1 -- 172.21.15.49:0/3912832424 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2368009320 con 0x7f236c0f6510 2023-11-30T21:46:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.933+0000 7f2373938700 1 -- 172.21.15.49:0/3912832424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236c106e20 con 0x7f236c0f6510 2023-11-30T21:46:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.933+0000 7f2361ffb700 1 -- 172.21.15.49:0/3912832424 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2368013f00 con 0x7f236c0f6510 2023-11-30T21:46:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.933+0000 7f2361ffb700 1 -- 172.21.15.49:0/3912832424 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2368014d90 con 0x7f236c0f6510 2023-11-30T21:46:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.933+0000 7f2373938700 1 -- 172.21.15.49:0/3912832424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236c1072b0 con 0x7f236c0f6510 2023-11-30T21:46:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.934+0000 7f2361ffb700 1 -- 172.21.15.49:0/3912832424 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2368014420 con 0x7f236c0f6510 2023-11-30T21:46:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.934+0000 7f2373938700 1 -- 172.21.15.49:0/3912832424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236c045ad0 con 0x7f236c0f6510 2023-11-30T21:46:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.934+0000 7f2361ffb700 1 --2- 172.21.15.49:0/3912832424 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f235802ff60 0x7f2358032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.934+0000 7f2361ffb700 1 -- 172.21.15.49:0/3912832424 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2368041050 con 0x7f236c0f6510 2023-11-30T21:46:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.934+0000 7f2370ed3700 1 --2- 172.21.15.49:0/3912832424 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f235802ff60 0x7f2358032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.934+0000 7f2370ed3700 1 --2- 172.21.15.49:0/3912832424 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f235802ff60 0x7f2358032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f235c0096a0 tx=0x7f235c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:47.940 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:47.938+0000 7f2361ffb700 1 -- 172.21.15.49:0/3912832424 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23680095f0 con 0x7f236c0f6510 2023-11-30T21:46:48.162 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:48.161+0000 7f2373938700 1 -- 172.21.15.49:0/3912832424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f236c0fbf60 con 0x7f236c0f6510 2023-11-30T21:46:48.163 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:48.161+0000 7f2361ffb700 1 -- 172.21.15.49:0/3912832424 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f236801db70 con 0x7f236c0f6510 2023-11-30T21:46:48.163 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:48.163 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:48.164 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:48.163+0000 7f2373938700 1 -- 172.21.15.49:0/3912832424 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f235802ff60 msgr2=0x7f2358032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:48.165 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:48.163+0000 7f2373938700 1 --2- 172.21.15.49:0/3912832424 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f235802ff60 0x7f2358032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f235c0096a0 tx=0x7f235c006b40).stop 2023-11-30T21:46:48.165 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:48.163+0000 7f2373938700 1 -- 172.21.15.49:0/3912832424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c0f6510 msgr2=0x7f236c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:48.165 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:48.163+0000 7f2373938700 1 --2- 172.21.15.49:0/3912832424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c0f6510 0x7f236c106730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f2368013510 tx=0x7f236800b040).stop 2023-11-30T21:46:48.165 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:48.163+0000 7f2373938700 1 -- 172.21.15.49:0/3912832424 shutdown_connections 2023-11-30T21:46:48.165 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:48.163+0000 7f2373938700 1 --2- 172.21.15.49:0/3912832424 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f235802ff60 0x7f2358032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:48.165 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:48.163+0000 7f2373938700 1 --2- 172.21.15.49:0/3912832424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f236c0f6510 0x7f236c106730 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:48.165 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:48.163+0000 7f2373938700 1 -- 172.21.15.49:0/3912832424 >> 172.21.15.49:0/3912832424 conn(0x7f236c0f1c70 msgr2=0x7f236c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:48.165 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:48.163+0000 7f2373938700 1 -- 172.21.15.49:0/3912832424 shutdown_connections 2023-11-30T21:46:48.165 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:48.163+0000 7f2373938700 1 -- 172.21.15.49:0/3912832424 wait complete. 2023-11-30T21:46:48.166 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:48.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:48 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:48.163041+0000 mon.smithi027 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.49:0/3912832424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:49.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:49 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:47.481900+0000 mgr.smithi027.brrjcl (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:49.539 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:49.539 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:50.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:50 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:49.482284+0000 mgr.smithi027.brrjcl (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:50.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.715+0000 7fb50265b700 1 -- 172.21.15.49:0/3572059849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4fc0f6480 msgr2=0x7fb4fc0f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:50.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.715+0000 7fb50265b700 1 --2- 172.21.15.49:0/3572059849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4fc0f6480 0x7fb4fc0f68a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb4ec009fb0 tx=0x7fb4ec009b30).stop 2023-11-30T21:46:50.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.715+0000 7fb50265b700 1 -- 172.21.15.49:0/3572059849 shutdown_connections 2023-11-30T21:46:50.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.715+0000 7fb50265b700 1 --2- 172.21.15.49:0/3572059849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4fc0f6480 0x7fb4fc0f68a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:50.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.715+0000 7fb50265b700 1 -- 172.21.15.49:0/3572059849 >> 172.21.15.49:0/3572059849 conn(0x7fb4fc0f1c00 msgr2=0x7fb4fc0f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:50.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.716+0000 7fb50265b700 1 -- 172.21.15.49:0/3572059849 shutdown_connections 2023-11-30T21:46:50.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.716+0000 7fb50265b700 1 -- 172.21.15.49:0/3572059849 wait complete. 2023-11-30T21:46:50.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.716+0000 7fb50265b700 1 Processor -- start 2023-11-30T21:46:50.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.716+0000 7fb50265b700 1 -- start start 2023-11-30T21:46:50.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.716+0000 7fb50265b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4fc0f6480 0x7fb4fc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:50.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.716+0000 7fb50265b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc069e20 con 0x7fb4fc0f6480 2023-11-30T21:46:50.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.717+0000 7fb4fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4fc0f6480 0x7fb4fc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:50.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.717+0000 7fb4fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4fc0f6480 0x7fb4fc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:37834/0 (socket says 172.21.15.49:37834) 2023-11-30T21:46:50.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.717+0000 7fb4fbfff700 1 -- 172.21.15.49:0/3944435201 learned_addr learned my addr 172.21.15.49:0/3944435201 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:50.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.717+0000 7fb4fbfff700 1 -- 172.21.15.49:0/3944435201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec005040 con 0x7fb4fc0f6480 2023-11-30T21:46:50.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.717+0000 7fb4fbfff700 1 --2- 172.21.15.49:0/3944435201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4fc0f6480 0x7fb4fc06b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fb4ec013510 tx=0x7fb4ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:50.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.717+0000 7fb4f8ff9700 1 -- 172.21.15.49:0/3944435201 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4ec009320 con 0x7fb4fc0f6480 2023-11-30T21:46:50.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.717+0000 7fb4f8ff9700 1 -- 172.21.15.49:0/3944435201 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4ec013f00 con 0x7fb4fc0f6480 2023-11-30T21:46:50.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.717+0000 7fb50265b700 1 -- 172.21.15.49:0/3944435201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4fc069fc0 con 0x7fb4fc0f6480 2023-11-30T21:46:50.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.717+0000 7fb4f8ff9700 1 -- 172.21.15.49:0/3944435201 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4ec014d90 con 0x7fb4fc0f6480 2023-11-30T21:46:50.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.718+0000 7fb50265b700 1 -- 172.21.15.49:0/3944435201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4fc06a450 con 0x7fb4fc0f6480 2023-11-30T21:46:50.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.718+0000 7fb4f8ff9700 1 -- 172.21.15.49:0/3944435201 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4ec0095f0 con 0x7fb4fc0f6480 2023-11-30T21:46:50.727 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.718+0000 7fb50265b700 1 -- 172.21.15.49:0/3944435201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4fc059100 con 0x7fb4fc0f6480 2023-11-30T21:46:50.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.718+0000 7fb4f8ff9700 1 --2- 172.21.15.49:0/3944435201 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb4e402ff60 0x7fb4e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:50.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.719+0000 7fb4f8ff9700 1 -- 172.21.15.49:0/3944435201 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4ec0410a0 con 0x7fb4fc0f6480 2023-11-30T21:46:50.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.719+0000 7fb4fb7fe700 1 --2- 172.21.15.49:0/3944435201 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb4e402ff60 0x7fb4e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:50.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.719+0000 7fb4fb7fe700 1 --2- 172.21.15.49:0/3944435201 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb4e402ff60 0x7fb4e4032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fb4f00096a0 tx=0x7fb4f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:50.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.723+0000 7fb4f8ff9700 1 -- 172.21.15.49:0/3944435201 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4ec01d7f0 con 0x7fb4fc0f6480 2023-11-30T21:46:50.943 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.941+0000 7fb50265b700 1 -- 172.21.15.49:0/3944435201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4fc0fbf40 con 0x7fb4fc0f6480 2023-11-30T21:46:50.943 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.942+0000 7fb4f8ff9700 1 -- 172.21.15.49:0/3944435201 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb4ec010230 con 0x7fb4fc0f6480 2023-11-30T21:46:50.943 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:50.943 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:50.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.943+0000 7fb50265b700 1 -- 172.21.15.49:0/3944435201 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb4e402ff60 msgr2=0x7fb4e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:50.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.943+0000 7fb50265b700 1 --2- 172.21.15.49:0/3944435201 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb4e402ff60 0x7fb4e4032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fb4f00096a0 tx=0x7fb4f0006b40).stop 2023-11-30T21:46:50.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.943+0000 7fb50265b700 1 -- 172.21.15.49:0/3944435201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4fc0f6480 msgr2=0x7fb4fc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:50.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.943+0000 7fb50265b700 1 --2- 172.21.15.49:0/3944435201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4fc0f6480 0x7fb4fc06b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fb4ec013510 tx=0x7fb4ec00b040).stop 2023-11-30T21:46:50.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.943+0000 7fb50265b700 1 -- 172.21.15.49:0/3944435201 shutdown_connections 2023-11-30T21:46:50.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.943+0000 7fb50265b700 1 --2- 172.21.15.49:0/3944435201 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb4e402ff60 0x7fb4e4032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:50.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.943+0000 7fb50265b700 1 --2- 172.21.15.49:0/3944435201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4fc0f6480 0x7fb4fc06b9a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:50.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.943+0000 7fb50265b700 1 -- 172.21.15.49:0/3944435201 >> 172.21.15.49:0/3944435201 conn(0x7fb4fc0f1c00 msgr2=0x7fb4fc0f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:50.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.943+0000 7fb50265b700 1 -- 172.21.15.49:0/3944435201 shutdown_connections 2023-11-30T21:46:50.945 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:50.943+0000 7fb50265b700 1 -- 172.21.15.49:0/3944435201 wait complete. 2023-11-30T21:46:50.946 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:51.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:50.943281+0000 mon.smithi027 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.49:0/3944435201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:52.326 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:52.326 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:52.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:52 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:51.482703+0000 mgr.smithi027.brrjcl (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:53.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.503+0000 7f79870f4700 1 -- 172.21.15.49:0/1671558688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79800f5410 msgr2=0x7f79800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:53.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.503+0000 7f79870f4700 1 --2- 172.21.15.49:0/1671558688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79800f5410 0x7f79800f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f7974009fb0 tx=0x7f7974009b30).stop 2023-11-30T21:46:53.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.503+0000 7f79870f4700 1 -- 172.21.15.49:0/1671558688 shutdown_connections 2023-11-30T21:46:53.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.503+0000 7f79870f4700 1 --2- 172.21.15.49:0/1671558688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79800f5410 0x7f79800f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:53.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.503+0000 7f79870f4700 1 -- 172.21.15.49:0/1671558688 >> 172.21.15.49:0/1671558688 conn(0x7f79800f0b70 msgr2=0x7f79800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:53.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.503+0000 7f79870f4700 1 -- 172.21.15.49:0/1671558688 shutdown_connections 2023-11-30T21:46:53.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.503+0000 7f79870f4700 1 -- 172.21.15.49:0/1671558688 wait complete. 2023-11-30T21:46:53.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.503+0000 7f79870f4700 1 Processor -- start 2023-11-30T21:46:53.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.503+0000 7f79870f4700 1 -- start start 2023-11-30T21:46:53.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.504+0000 7f79870f4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79800f5410 0x7f7980105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:53.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.504+0000 7f79870f4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7980105b20 con 0x7f79800f5410 2023-11-30T21:46:53.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.504+0000 7f7984e90700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79800f5410 0x7f7980105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:53.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.504+0000 7f7984e90700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79800f5410 0x7f7980105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:37858/0 (socket says 172.21.15.49:37858) 2023-11-30T21:46:53.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.504+0000 7f7984e90700 1 -- 172.21.15.49:0/1556731879 learned_addr learned my addr 172.21.15.49:0/1556731879 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:53.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.504+0000 7f7984e90700 1 -- 172.21.15.49:0/1556731879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7974005040 con 0x7f79800f5410 2023-11-30T21:46:53.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.504+0000 7f7984e90700 1 --2- 172.21.15.49:0/1556731879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79800f5410 0x7f7980105630 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f7974013510 tx=0x7f797400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:53.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.505+0000 7f797d7fa700 1 -- 172.21.15.49:0/1556731879 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7974009320 con 0x7f79800f5410 2023-11-30T21:46:53.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.505+0000 7f797d7fa700 1 -- 172.21.15.49:0/1556731879 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7974013f00 con 0x7f79800f5410 2023-11-30T21:46:53.508 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.505+0000 7f79870f4700 1 -- 172.21.15.49:0/1556731879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7980105d20 con 0x7f79800f5410 2023-11-30T21:46:53.508 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.505+0000 7f797d7fa700 1 -- 172.21.15.49:0/1556731879 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7974014d90 con 0x7f79800f5410 2023-11-30T21:46:53.508 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.505+0000 7f79870f4700 1 -- 172.21.15.49:0/1556731879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79801061b0 con 0x7f79800f5410 2023-11-30T21:46:53.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.505+0000 7f797d7fa700 1 -- 172.21.15.49:0/1556731879 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7974014420 con 0x7f79800f5410 2023-11-30T21:46:53.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.505+0000 7f79870f4700 1 -- 172.21.15.49:0/1556731879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7980045ad0 con 0x7f79800f5410 2023-11-30T21:46:53.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.506+0000 7f797d7fa700 1 --2- 172.21.15.49:0/1556731879 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f797002ff60 0x7f7970032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:53.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.506+0000 7f797d7fa700 1 -- 172.21.15.49:0/1556731879 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7974040ff0 con 0x7f79800f5410 2023-11-30T21:46:53.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.506+0000 7f797ffff700 1 --2- 172.21.15.49:0/1556731879 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f797002ff60 0x7f7970032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:53.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.506+0000 7f797ffff700 1 --2- 172.21.15.49:0/1556731879 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f797002ff60 0x7f7970032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f796c0096a0 tx=0x7f796c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:53.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.511+0000 7f797d7fa700 1 -- 172.21.15.49:0/1556731879 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f79740095f0 con 0x7f79800f5410 2023-11-30T21:46:53.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.729+0000 7f79870f4700 1 -- 172.21.15.49:0/1556731879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79800fae60 con 0x7f79800f5410 2023-11-30T21:46:53.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.730+0000 7f797d7fa700 1 -- 172.21.15.49:0/1556731879 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f797401db70 con 0x7f79800f5410 2023-11-30T21:46:53.731 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:53.731 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:53.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.731+0000 7f79870f4700 1 -- 172.21.15.49:0/1556731879 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f797002ff60 msgr2=0x7f7970032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:53.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.731+0000 7f79870f4700 1 --2- 172.21.15.49:0/1556731879 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f797002ff60 0x7f7970032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f796c0096a0 tx=0x7f796c006b40).stop 2023-11-30T21:46:53.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.731+0000 7f79870f4700 1 -- 172.21.15.49:0/1556731879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79800f5410 msgr2=0x7f7980105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:53.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.731+0000 7f79870f4700 1 --2- 172.21.15.49:0/1556731879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79800f5410 0x7f7980105630 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f7974013510 tx=0x7f797400b040).stop 2023-11-30T21:46:53.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.731+0000 7f79870f4700 1 -- 172.21.15.49:0/1556731879 shutdown_connections 2023-11-30T21:46:53.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.731+0000 7f79870f4700 1 --2- 172.21.15.49:0/1556731879 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f797002ff60 0x7f7970032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:53.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.731+0000 7f79870f4700 1 --2- 172.21.15.49:0/1556731879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79800f5410 0x7f7980105630 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:53.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.731+0000 7f79870f4700 1 -- 172.21.15.49:0/1556731879 >> 172.21.15.49:0/1556731879 conn(0x7f79800f0b70 msgr2=0x7f79800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:53.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.731+0000 7f79870f4700 1 -- 172.21.15.49:0/1556731879 shutdown_connections 2023-11-30T21:46:53.734 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:53.731+0000 7f79870f4700 1 -- 172.21.15.49:0/1556731879 wait complete. 2023-11-30T21:46:53.734 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:55.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:54 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:53.483058+0000 mgr.smithi027.brrjcl (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:55.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:54 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:53.731246+0000 mon.smithi027 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.49:0/1556731879' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:55.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:55.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:56.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.254+0000 7f9d84495700 1 -- 172.21.15.49:0/3655461309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d7c0f5410 msgr2=0x7f9d7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:56.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.254+0000 7f9d84495700 1 --2- 172.21.15.49:0/3655461309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d7c0f5410 0x7f9d7c0f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f9d78009fb0 tx=0x7f9d78009b30).stop 2023-11-30T21:46:56.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.254+0000 7f9d84495700 1 -- 172.21.15.49:0/3655461309 shutdown_connections 2023-11-30T21:46:56.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.254+0000 7f9d84495700 1 --2- 172.21.15.49:0/3655461309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d7c0f5410 0x7f9d7c0f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:56.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.254+0000 7f9d84495700 1 -- 172.21.15.49:0/3655461309 >> 172.21.15.49:0/3655461309 conn(0x7f9d7c0f0b70 msgr2=0x7f9d7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:56.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.254+0000 7f9d84495700 1 -- 172.21.15.49:0/3655461309 shutdown_connections 2023-11-30T21:46:56.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.254+0000 7f9d84495700 1 -- 172.21.15.49:0/3655461309 wait complete. 2023-11-30T21:46:56.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.255+0000 7f9d84495700 1 Processor -- start 2023-11-30T21:46:56.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.255+0000 7f9d84495700 1 -- start start 2023-11-30T21:46:56.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.255+0000 7f9d84495700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d7c0f5410 0x7f9d7c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:56.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.255+0000 7f9d84495700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c106c10 con 0x7f9d7c0f5410 2023-11-30T21:46:56.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.255+0000 7f9d82231700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d7c0f5410 0x7f9d7c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:56.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.255+0000 7f9d82231700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d7c0f5410 0x7f9d7c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:37878/0 (socket says 172.21.15.49:37878) 2023-11-30T21:46:56.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.255+0000 7f9d82231700 1 -- 172.21.15.49:0/1818412643 learned_addr learned my addr 172.21.15.49:0/1818412643 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:56.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.256+0000 7f9d82231700 1 -- 172.21.15.49:0/1818412643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d78005040 con 0x7f9d7c0f5410 2023-11-30T21:46:56.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.256+0000 7f9d82231700 1 --2- 172.21.15.49:0/1818412643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d7c0f5410 0x7f9d7c106720 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9d78013510 tx=0x7f9d7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:56.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.256+0000 7f9d6effd700 1 -- 172.21.15.49:0/1818412643 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d78009320 con 0x7f9d7c0f5410 2023-11-30T21:46:56.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.256+0000 7f9d84495700 1 -- 172.21.15.49:0/1818412643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7c106e10 con 0x7f9d7c0f5410 2023-11-30T21:46:56.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.256+0000 7f9d6effd700 1 -- 172.21.15.49:0/1818412643 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d78013f00 con 0x7f9d7c0f5410 2023-11-30T21:46:56.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.256+0000 7f9d6effd700 1 -- 172.21.15.49:0/1818412643 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d78014d90 con 0x7f9d7c0f5410 2023-11-30T21:46:56.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.256+0000 7f9d84495700 1 -- 172.21.15.49:0/1818412643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7c1072a0 con 0x7f9d7c0f5410 2023-11-30T21:46:56.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.257+0000 7f9d6effd700 1 -- 172.21.15.49:0/1818412643 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d780095f0 con 0x7f9d7c0f5410 2023-11-30T21:46:56.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.257+0000 7f9d84495700 1 -- 172.21.15.49:0/1818412643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d7c045ad0 con 0x7f9d7c0f5410 2023-11-30T21:46:56.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.257+0000 7f9d6effd700 1 --2- 172.21.15.49:0/1818412643 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d6802ffb0 0x7f9d68032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:56.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.257+0000 7f9d6effd700 1 -- 172.21.15.49:0/1818412643 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d78040fe0 con 0x7f9d7c0f5410 2023-11-30T21:46:56.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.257+0000 7f9d81a30700 1 --2- 172.21.15.49:0/1818412643 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d6802ffb0 0x7f9d68032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:56.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.258+0000 7f9d81a30700 1 --2- 172.21.15.49:0/1818412643 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d6802ffb0 0x7f9d68032470 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9d700096a0 tx=0x7f9d70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:56.262 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.262+0000 7f9d6effd700 1 -- 172.21.15.49:0/1818412643 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d7806d2b0 con 0x7f9d7c0f5410 2023-11-30T21:46:56.480 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.479+0000 7f9d84495700 1 -- 172.21.15.49:0/1818412643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d7c108150 con 0x7f9d7c0f5410 2023-11-30T21:46:56.480 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.480+0000 7f9d6effd700 1 -- 172.21.15.49:0/1818412643 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9d78014420 con 0x7f9d7c0f5410 2023-11-30T21:46:56.481 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:56.481 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:56.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.482+0000 7f9d84495700 1 -- 172.21.15.49:0/1818412643 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d6802ffb0 msgr2=0x7f9d68032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:56.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.482+0000 7f9d84495700 1 --2- 172.21.15.49:0/1818412643 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d6802ffb0 0x7f9d68032470 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9d700096a0 tx=0x7f9d70006b40).stop 2023-11-30T21:46:56.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.482+0000 7f9d84495700 1 -- 172.21.15.49:0/1818412643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d7c0f5410 msgr2=0x7f9d7c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:56.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.482+0000 7f9d84495700 1 --2- 172.21.15.49:0/1818412643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d7c0f5410 0x7f9d7c106720 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9d78013510 tx=0x7f9d7800b040).stop 2023-11-30T21:46:56.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.482+0000 7f9d84495700 1 -- 172.21.15.49:0/1818412643 shutdown_connections 2023-11-30T21:46:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.482+0000 7f9d84495700 1 --2- 172.21.15.49:0/1818412643 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d6802ffb0 0x7f9d68032470 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.482+0000 7f9d84495700 1 --2- 172.21.15.49:0/1818412643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d7c0f5410 0x7f9d7c106720 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.482+0000 7f9d84495700 1 -- 172.21.15.49:0/1818412643 >> 172.21.15.49:0/1818412643 conn(0x7f9d7c0f0b70 msgr2=0x7f9d7c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.482+0000 7f9d84495700 1 -- 172.21.15.49:0/1818412643 shutdown_connections 2023-11-30T21:46:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:56.482+0000 7f9d84495700 1 -- 172.21.15.49:0/1818412643 wait complete. 2023-11-30T21:46:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:46:57.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:56 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:55.483447+0000 mgr.smithi027.brrjcl (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:57.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:56 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:56.480434+0000 mon.smithi027 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.49:0/1818412643' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:46:57.858 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:46:57.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:46:59.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:46:58 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:57.483866+0000 mgr.smithi027.brrjcl (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:46:59.020 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.019+0000 7f7255143700 1 -- 172.21.15.49:0/634409233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72500f4740 msgr2=0x7f72500f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:59.020 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.019+0000 7f7255143700 1 --2- 172.21.15.49:0/634409233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72500f4740 0x7f72500f4b60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f7240009fb0 tx=0x7f7240009b30).stop 2023-11-30T21:46:59.020 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.019+0000 7f7255143700 1 -- 172.21.15.49:0/634409233 shutdown_connections 2023-11-30T21:46:59.020 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.019+0000 7f7255143700 1 --2- 172.21.15.49:0/634409233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72500f4740 0x7f72500f4b60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:59.020 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.019+0000 7f7255143700 1 -- 172.21.15.49:0/634409233 >> 172.21.15.49:0/634409233 conn(0x7f72500f0260 msgr2=0x7f72500f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:59.020 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.019+0000 7f7255143700 1 -- 172.21.15.49:0/634409233 shutdown_connections 2023-11-30T21:46:59.020 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.019+0000 7f7255143700 1 -- 172.21.15.49:0/634409233 wait complete. 2023-11-30T21:46:59.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.020+0000 7f7255143700 1 Processor -- start 2023-11-30T21:46:59.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.020+0000 7f7255143700 1 -- start start 2023-11-30T21:46:59.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.020+0000 7f7255143700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72500f4740 0x7f72501066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:59.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.020+0000 7f7255143700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7250106bc0 con 0x7f72500f4740 2023-11-30T21:46:59.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.020+0000 7f724ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72500f4740 0x7f72501066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:59.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.020+0000 7f724ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72500f4740 0x7f72501066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:34244/0 (socket says 172.21.15.49:34244) 2023-11-30T21:46:59.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.020+0000 7f724ed9d700 1 -- 172.21.15.49:0/748216985 learned_addr learned my addr 172.21.15.49:0/748216985 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:46:59.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.020+0000 7f724ed9d700 1 -- 172.21.15.49:0/748216985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7240005040 con 0x7f72500f4740 2023-11-30T21:46:59.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.021+0000 7f724ed9d700 1 --2- 172.21.15.49:0/748216985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72500f4740 0x7f72501066d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f7240013510 tx=0x7f724000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:59.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.021+0000 7f72477fe700 1 -- 172.21.15.49:0/748216985 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7240009320 con 0x7f72500f4740 2023-11-30T21:46:59.023 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.021+0000 7f7255143700 1 -- 172.21.15.49:0/748216985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7250106dc0 con 0x7f72500f4740 2023-11-30T21:46:59.023 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.021+0000 7f72477fe700 1 -- 172.21.15.49:0/748216985 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7240013f00 con 0x7f72500f4740 2023-11-30T21:46:59.023 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.021+0000 7f72477fe700 1 -- 172.21.15.49:0/748216985 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7240014d90 con 0x7f72500f4740 2023-11-30T21:46:59.023 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.021+0000 7f7255143700 1 -- 172.21.15.49:0/748216985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7250107250 con 0x7f72500f4740 2023-11-30T21:46:59.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.022+0000 7f72477fe700 1 -- 172.21.15.49:0/748216985 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72400095f0 con 0x7f72500f4740 2023-11-30T21:46:59.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.022+0000 7f7255143700 1 -- 172.21.15.49:0/748216985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7250045ad0 con 0x7f72500f4740 2023-11-30T21:46:59.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.022+0000 7f72477fe700 1 --2- 172.21.15.49:0/748216985 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f723c02ff60 0x7f723c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:46:59.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.022+0000 7f72477fe700 1 -- 172.21.15.49:0/748216985 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7240041000 con 0x7f72500f4740 2023-11-30T21:46:59.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.022+0000 7f724e59c700 1 --2- 172.21.15.49:0/748216985 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f723c02ff60 0x7f723c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:46:59.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.022+0000 7f724e59c700 1 --2- 172.21.15.49:0/748216985 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f723c02ff60 0x7f723c032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f72380096a0 tx=0x7f7238006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:46:59.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.027+0000 7f72477fe700 1 -- 172.21.15.49:0/748216985 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7240014420 con 0x7f72500f4740 2023-11-30T21:46:59.247 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.246+0000 7f7255143700 1 -- 172.21.15.49:0/748216985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7250108100 con 0x7f72500f4740 2023-11-30T21:46:59.248 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.247+0000 7f72477fe700 1 -- 172.21.15.49:0/748216985 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f724001d7f0 con 0x7f72500f4740 2023-11-30T21:46:59.248 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:46:59.248 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:46:59.249 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.249+0000 7f7255143700 1 -- 172.21.15.49:0/748216985 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f723c02ff60 msgr2=0x7f723c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:59.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.249+0000 7f7255143700 1 --2- 172.21.15.49:0/748216985 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f723c02ff60 0x7f723c032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f72380096a0 tx=0x7f7238006b40).stop 2023-11-30T21:46:59.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.249+0000 7f7255143700 1 -- 172.21.15.49:0/748216985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72500f4740 msgr2=0x7f72501066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:46:59.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.249+0000 7f7255143700 1 --2- 172.21.15.49:0/748216985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72500f4740 0x7f72501066d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f7240013510 tx=0x7f724000b040).stop 2023-11-30T21:46:59.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.249+0000 7f7255143700 1 -- 172.21.15.49:0/748216985 shutdown_connections 2023-11-30T21:46:59.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.249+0000 7f7255143700 1 --2- 172.21.15.49:0/748216985 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f723c02ff60 0x7f723c032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:59.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.249+0000 7f7255143700 1 --2- 172.21.15.49:0/748216985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72500f4740 0x7f72501066d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:46:59.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.249+0000 7f7255143700 1 -- 172.21.15.49:0/748216985 >> 172.21.15.49:0/748216985 conn(0x7f72500f0260 msgr2=0x7f72500f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:46:59.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.249+0000 7f7255143700 1 -- 172.21.15.49:0/748216985 shutdown_connections 2023-11-30T21:46:59.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:46:59.249+0000 7f7255143700 1 -- 172.21.15.49:0/748216985 wait complete. 2023-11-30T21:46:59.251 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:00.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:00 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:46:59.247559+0000 mon.smithi027 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.49:0/748216985' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:00.626 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:00.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:01.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:01 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:46:59.484299+0000 mgr.smithi027.brrjcl (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:01.800 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.799+0000 7feb6a00b700 1 -- 172.21.15.49:0/4101960824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb640f5410 msgr2=0x7feb640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:01.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.799+0000 7feb6a00b700 1 --2- 172.21.15.49:0/4101960824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb640f5410 0x7feb640f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7feb54009fb0 tx=0x7feb54009b30).stop 2023-11-30T21:47:01.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.799+0000 7feb6a00b700 1 -- 172.21.15.49:0/4101960824 shutdown_connections 2023-11-30T21:47:01.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.799+0000 7feb6a00b700 1 --2- 172.21.15.49:0/4101960824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb640f5410 0x7feb640f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:01.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.799+0000 7feb6a00b700 1 -- 172.21.15.49:0/4101960824 >> 172.21.15.49:0/4101960824 conn(0x7feb640f0b70 msgr2=0x7feb640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:01.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.799+0000 7feb6a00b700 1 -- 172.21.15.49:0/4101960824 shutdown_connections 2023-11-30T21:47:01.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.799+0000 7feb6a00b700 1 -- 172.21.15.49:0/4101960824 wait complete. 2023-11-30T21:47:01.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.799+0000 7feb6a00b700 1 Processor -- start 2023-11-30T21:47:01.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.800+0000 7feb6a00b700 1 -- start start 2023-11-30T21:47:01.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.800+0000 7feb6a00b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb640f5410 0x7feb6410b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:01.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.800+0000 7feb6a00b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb641071e0 con 0x7feb640f5410 2023-11-30T21:47:01.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.800+0000 7feb637fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb640f5410 0x7feb6410b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:01.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.800+0000 7feb637fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb640f5410 0x7feb6410b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:34276/0 (socket says 172.21.15.49:34276) 2023-11-30T21:47:01.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.800+0000 7feb637fe700 1 -- 172.21.15.49:0/2206545248 learned_addr learned my addr 172.21.15.49:0/2206545248 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.800+0000 7feb637fe700 1 -- 172.21.15.49:0/2206545248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb54005040 con 0x7feb640f5410 2023-11-30T21:47:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.800+0000 7feb637fe700 1 --2- 172.21.15.49:0/2206545248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb640f5410 0x7feb6410b7f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7feb54013510 tx=0x7feb5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.801+0000 7feb5bfff700 1 -- 172.21.15.49:0/2206545248 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb54009320 con 0x7feb640f5410 2023-11-30T21:47:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.801+0000 7feb6a00b700 1 -- 172.21.15.49:0/2206545248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb64060280 con 0x7feb640f5410 2023-11-30T21:47:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.801+0000 7feb5bfff700 1 -- 172.21.15.49:0/2206545248 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb54013f00 con 0x7feb640f5410 2023-11-30T21:47:01.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.801+0000 7feb5bfff700 1 -- 172.21.15.49:0/2206545248 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb54014d90 con 0x7feb640f5410 2023-11-30T21:47:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.801+0000 7feb6a00b700 1 -- 172.21.15.49:0/2206545248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb6410bce0 con 0x7feb640f5410 2023-11-30T21:47:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.801+0000 7feb5bfff700 1 -- 172.21.15.49:0/2206545248 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb540095f0 con 0x7feb640f5410 2023-11-30T21:47:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.801+0000 7feb6a00b700 1 -- 172.21.15.49:0/2206545248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb64045ad0 con 0x7feb640f5410 2023-11-30T21:47:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.802+0000 7feb5bfff700 1 --2- 172.21.15.49:0/2206545248 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feb5002ff60 0x7feb50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.802+0000 7feb5bfff700 1 -- 172.21.15.49:0/2206545248 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb54041080 con 0x7feb640f5410 2023-11-30T21:47:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.802+0000 7feb62ffd700 1 --2- 172.21.15.49:0/2206545248 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feb5002ff60 0x7feb50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:01.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.802+0000 7feb62ffd700 1 --2- 172.21.15.49:0/2206545248 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feb5002ff60 0x7feb50032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7feb4c0096a0 tx=0x7feb4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:01.807 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:01.806+0000 7feb5bfff700 1 -- 172.21.15.49:0/2206545248 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb54014420 con 0x7feb640f5410 2023-11-30T21:47:02.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:02.025+0000 7feb6a00b700 1 -- 172.21.15.49:0/2206545248 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb640fae60 con 0x7feb640f5410 2023-11-30T21:47:02.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:02.026+0000 7feb5bfff700 1 -- 172.21.15.49:0/2206545248 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb54010230 con 0x7feb640f5410 2023-11-30T21:47:02.027 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:02.027 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:02.028+0000 7feb6a00b700 1 -- 172.21.15.49:0/2206545248 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feb5002ff60 msgr2=0x7feb50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:02.028+0000 7feb6a00b700 1 --2- 172.21.15.49:0/2206545248 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feb5002ff60 0x7feb50032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7feb4c0096a0 tx=0x7feb4c006b40).stop 2023-11-30T21:47:02.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:02.028+0000 7feb6a00b700 1 -- 172.21.15.49:0/2206545248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb640f5410 msgr2=0x7feb6410b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:02.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:02.028+0000 7feb6a00b700 1 --2- 172.21.15.49:0/2206545248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb640f5410 0x7feb6410b7f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7feb54013510 tx=0x7feb5400b040).stop 2023-11-30T21:47:02.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:02.028+0000 7feb6a00b700 1 -- 172.21.15.49:0/2206545248 shutdown_connections 2023-11-30T21:47:02.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:02.028+0000 7feb6a00b700 1 --2- 172.21.15.49:0/2206545248 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feb5002ff60 0x7feb50032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:02.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:02.028+0000 7feb6a00b700 1 --2- 172.21.15.49:0/2206545248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb640f5410 0x7feb6410b7f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:02.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:02.028+0000 7feb6a00b700 1 -- 172.21.15.49:0/2206545248 >> 172.21.15.49:0/2206545248 conn(0x7feb640f0b70 msgr2=0x7feb640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:02.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:02.028+0000 7feb6a00b700 1 -- 172.21.15.49:0/2206545248 shutdown_connections 2023-11-30T21:47:02.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:02.028+0000 7feb6a00b700 1 -- 172.21.15.49:0/2206545248 wait complete. 2023-11-30T21:47:02.029 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:02.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:02 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:02.026647+0000 mon.smithi027 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.49:0/2206545248' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:03.428 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:03.429 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:03.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:03 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:01.484670+0000 mgr.smithi027.brrjcl (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:03.585295+0000 mon.smithi027 (mon.0) 169 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/mirror_snapshot_schedule"}]: dispatch 2023-11-30T21:47:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:03.592818+0000 mon.smithi027 (mon.0) 170 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/trash_purge_schedule"}]: dispatch 2023-11-30T21:47:04.605 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.603+0000 7f4f5759e700 1 -- 172.21.15.49:0/1851021908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f580f51a0 msgr2=0x7f4f580f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:04.605 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.603+0000 7f4f5759e700 1 --2- 172.21.15.49:0/1851021908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f580f51a0 0x7f4f580f55c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f4f48009fb0 tx=0x7f4f48009b30).stop 2023-11-30T21:47:04.605 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.603+0000 7f4f5759e700 1 -- 172.21.15.49:0/1851021908 shutdown_connections 2023-11-30T21:47:04.605 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.603+0000 7f4f5759e700 1 --2- 172.21.15.49:0/1851021908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f580f51a0 0x7f4f580f55c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:04.605 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.603+0000 7f4f5759e700 1 -- 172.21.15.49:0/1851021908 >> 172.21.15.49:0/1851021908 conn(0x7f4f580f09a0 msgr2=0x7f4f580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:04.605 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.603+0000 7f4f5759e700 1 -- 172.21.15.49:0/1851021908 shutdown_connections 2023-11-30T21:47:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.603+0000 7f4f5759e700 1 -- 172.21.15.49:0/1851021908 wait complete. 2023-11-30T21:47:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.603+0000 7f4f5759e700 1 Processor -- start 2023-11-30T21:47:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.603+0000 7f4f5759e700 1 -- start start 2023-11-30T21:47:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.604+0000 7f4f5759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f580f51a0 0x7f4f580fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.604+0000 7f4f5759e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f580fb5a0 con 0x7f4f580f51a0 2023-11-30T21:47:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.604+0000 7f4f5659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f580f51a0 0x7f4f580fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.604+0000 7f4f5659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f580f51a0 0x7f4f580fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:34302/0 (socket says 172.21.15.49:34302) 2023-11-30T21:47:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.604+0000 7f4f5659c700 1 -- 172.21.15.49:0/1445674211 learned_addr learned my addr 172.21.15.49:0/1445674211 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.604+0000 7f4f5659c700 1 -- 172.21.15.49:0/1445674211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f48005040 con 0x7f4f580f51a0 2023-11-30T21:47:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.604+0000 7f4f5659c700 1 --2- 172.21.15.49:0/1445674211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f580f51a0 0x7f4f580fd120 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f4f48013510 tx=0x7f4f4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.605+0000 7f4f46ffd700 1 -- 172.21.15.49:0/1445674211 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f48009320 con 0x7f4f580f51a0 2023-11-30T21:47:04.607 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.605+0000 7f4f5759e700 1 -- 172.21.15.49:0/1445674211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f580fb740 con 0x7f4f580f51a0 2023-11-30T21:47:04.607 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.605+0000 7f4f46ffd700 1 -- 172.21.15.49:0/1445674211 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f48013f00 con 0x7f4f580f51a0 2023-11-30T21:47:04.607 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.605+0000 7f4f46ffd700 1 -- 172.21.15.49:0/1445674211 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f48014dc0 con 0x7f4f580f51a0 2023-11-30T21:47:04.607 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.605+0000 7f4f5759e700 1 -- 172.21.15.49:0/1445674211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f580fbbd0 con 0x7f4f580f51a0 2023-11-30T21:47:04.608 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.605+0000 7f4f46ffd700 1 -- 172.21.15.49:0/1445674211 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4f4801d3d0 con 0x7f4f580f51a0 2023-11-30T21:47:04.608 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.606+0000 7f4f5759e700 1 -- 172.21.15.49:0/1445674211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f58059100 con 0x7f4f580f51a0 2023-11-30T21:47:04.608 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.606+0000 7f4f46ffd700 1 --2- 172.21.15.49:0/1445674211 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4f4002ffb0 0x7f4f40032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:04.608 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.606+0000 7f4f46ffd700 1 -- 172.21.15.49:0/1445674211 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4f48041110 con 0x7f4f580f51a0 2023-11-30T21:47:04.608 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.606+0000 7f4f55d9b700 1 --2- 172.21.15.49:0/1445674211 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4f4002ffb0 0x7f4f40032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:04.608 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.606+0000 7f4f55d9b700 1 --2- 172.21.15.49:0/1445674211 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4f4002ffb0 0x7f4f40032470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f4f4c0096a0 tx=0x7f4f4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:04.611 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.610+0000 7f4f46ffd700 1 -- 172.21.15.49:0/1445674211 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f4801cb10 con 0x7f4f580f51a0 2023-11-30T21:47:04.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.832+0000 7f4f5759e700 1 -- 172.21.15.49:0/1445674211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f580fabe0 con 0x7f4f580f51a0 2023-11-30T21:47:04.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.832+0000 7f4f46ffd700 1 -- 172.21.15.49:0/1445674211 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f4801cb10 con 0x7f4f580f51a0 2023-11-30T21:47:04.833 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:04.833 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:04.835 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.834+0000 7f4f5759e700 1 -- 172.21.15.49:0/1445674211 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4f4002ffb0 msgr2=0x7f4f40032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:04.835 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.834+0000 7f4f5759e700 1 --2- 172.21.15.49:0/1445674211 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4f4002ffb0 0x7f4f40032470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f4f4c0096a0 tx=0x7f4f4c006b40).stop 2023-11-30T21:47:04.835 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.834+0000 7f4f5759e700 1 -- 172.21.15.49:0/1445674211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f580f51a0 msgr2=0x7f4f580fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:04.835 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.834+0000 7f4f5759e700 1 --2- 172.21.15.49:0/1445674211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f580f51a0 0x7f4f580fd120 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f4f48013510 tx=0x7f4f4800b040).stop 2023-11-30T21:47:04.835 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.834+0000 7f4f5759e700 1 -- 172.21.15.49:0/1445674211 shutdown_connections 2023-11-30T21:47:04.836 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.834+0000 7f4f5759e700 1 --2- 172.21.15.49:0/1445674211 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4f4002ffb0 0x7f4f40032470 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:04.836 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.834+0000 7f4f5759e700 1 --2- 172.21.15.49:0/1445674211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f580f51a0 0x7f4f580fd120 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:04.836 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.834+0000 7f4f5759e700 1 -- 172.21.15.49:0/1445674211 >> 172.21.15.49:0/1445674211 conn(0x7f4f580f09a0 msgr2=0x7f4f5805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:04.836 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.834+0000 7f4f5759e700 1 -- 172.21.15.49:0/1445674211 shutdown_connections 2023-11-30T21:47:04.836 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:04.834+0000 7f4f5759e700 1 -- 172.21.15.49:0/1445674211 wait complete. 2023-11-30T21:47:04.836 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:05.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:03.485125+0000 mgr.smithi027.brrjcl (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:05.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:04.832765+0000 mon.smithi027 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.49:0/1445674211' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:06.263 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:06.264 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:07.500 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.495+0000 7f231ba36700 1 -- 172.21.15.49:0/4167637317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2314060280 msgr2=0x7f23140f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:07.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.495+0000 7f231ba36700 1 --2- 172.21.15.49:0/4167637317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2314060280 0x7f23140f90f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f2310009fb0 tx=0x7f2310009b30).stop 2023-11-30T21:47:07.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.495+0000 7f231ba36700 1 -- 172.21.15.49:0/4167637317 shutdown_connections 2023-11-30T21:47:07.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.495+0000 7f231ba36700 1 --2- 172.21.15.49:0/4167637317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2314060280 0x7f23140f90f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:07.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.495+0000 7f231ba36700 1 -- 172.21.15.49:0/4167637317 >> 172.21.15.49:0/4167637317 conn(0x7f23140f0b70 msgr2=0x7f23140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:07.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.495+0000 7f231ba36700 1 -- 172.21.15.49:0/4167637317 shutdown_connections 2023-11-30T21:47:07.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.495+0000 7f231ba36700 1 -- 172.21.15.49:0/4167637317 wait complete. 2023-11-30T21:47:07.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.496+0000 7f231ba36700 1 Processor -- start 2023-11-30T21:47:07.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.496+0000 7f231ba36700 1 -- start start 2023-11-30T21:47:07.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.496+0000 7f231ba36700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2314060280 0x7f23141066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:07.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.496+0000 7f231ba36700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2314106bd0 con 0x7f2314060280 2023-11-30T21:47:07.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.496+0000 7f23197d2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2314060280 0x7f23141066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:07.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.496+0000 7f23197d2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2314060280 0x7f23141066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:34324/0 (socket says 172.21.15.49:34324) 2023-11-30T21:47:07.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.496+0000 7f23197d2700 1 -- 172.21.15.49:0/1463311552 learned_addr learned my addr 172.21.15.49:0/1463311552 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:07.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.497+0000 7f23197d2700 1 -- 172.21.15.49:0/1463311552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2310005040 con 0x7f2314060280 2023-11-30T21:47:07.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.497+0000 7f23197d2700 1 --2- 172.21.15.49:0/1463311552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2314060280 0x7f23141066e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f2310013510 tx=0x7f231000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:07.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.497+0000 7f2305ffb700 1 -- 172.21.15.49:0/1463311552 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2310009320 con 0x7f2314060280 2023-11-30T21:47:07.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.497+0000 7f231ba36700 1 -- 172.21.15.49:0/1463311552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2314106dd0 con 0x7f2314060280 2023-11-30T21:47:07.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.497+0000 7f2305ffb700 1 -- 172.21.15.49:0/1463311552 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2310013f00 con 0x7f2314060280 2023-11-30T21:47:07.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.497+0000 7f2305ffb700 1 -- 172.21.15.49:0/1463311552 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2310014dc0 con 0x7f2314060280 2023-11-30T21:47:07.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.497+0000 7f231ba36700 1 -- 172.21.15.49:0/1463311552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2314107260 con 0x7f2314060280 2023-11-30T21:47:07.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.498+0000 7f2305ffb700 1 -- 172.21.15.49:0/1463311552 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23100095f0 con 0x7f2314060280 2023-11-30T21:47:07.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.498+0000 7f231ba36700 1 -- 172.21.15.49:0/1463311552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2314045ad0 con 0x7f2314060280 2023-11-30T21:47:07.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.498+0000 7f2305ffb700 1 --2- 172.21.15.49:0/1463311552 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f230002ff60 0x7f2300032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:07.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.498+0000 7f2305ffb700 1 -- 172.21.15.49:0/1463311552 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2310041040 con 0x7f2314060280 2023-11-30T21:47:07.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.499+0000 7f2318fd1700 1 --2- 172.21.15.49:0/1463311552 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f230002ff60 0x7f2300032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:07.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.499+0000 7f2318fd1700 1 --2- 172.21.15.49:0/1463311552 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f230002ff60 0x7f2300032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f23080096a0 tx=0x7f2308006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:07.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.503+0000 7f2305ffb700 1 -- 172.21.15.49:0/1463311552 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2310014420 con 0x7f2314060280 2023-11-30T21:47:07.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:07 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:05.485447+0000 mgr.smithi027.brrjcl (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:07.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.720+0000 7f231ba36700 1 -- 172.21.15.49:0/1463311552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2314108110 con 0x7f2314060280 2023-11-30T21:47:07.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.721+0000 7f2305ffb700 1 -- 172.21.15.49:0/1463311552 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2310010230 con 0x7f2314060280 2023-11-30T21:47:07.722 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:07.723 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:07.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.722+0000 7f231ba36700 1 -- 172.21.15.49:0/1463311552 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f230002ff60 msgr2=0x7f2300032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:07.724 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.722+0000 7f231ba36700 1 --2- 172.21.15.49:0/1463311552 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f230002ff60 0x7f2300032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f23080096a0 tx=0x7f2308006b40).stop 2023-11-30T21:47:07.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.722+0000 7f231ba36700 1 -- 172.21.15.49:0/1463311552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2314060280 msgr2=0x7f23141066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:07.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.722+0000 7f231ba36700 1 --2- 172.21.15.49:0/1463311552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2314060280 0x7f23141066e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f2310013510 tx=0x7f231000b040).stop 2023-11-30T21:47:07.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.722+0000 7f231ba36700 1 -- 172.21.15.49:0/1463311552 shutdown_connections 2023-11-30T21:47:07.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.722+0000 7f231ba36700 1 --2- 172.21.15.49:0/1463311552 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f230002ff60 0x7f2300032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:07.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.722+0000 7f231ba36700 1 --2- 172.21.15.49:0/1463311552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2314060280 0x7f23141066e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:07.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.722+0000 7f231ba36700 1 -- 172.21.15.49:0/1463311552 >> 172.21.15.49:0/1463311552 conn(0x7f23140f0b70 msgr2=0x7f23140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:07.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.722+0000 7f231ba36700 1 -- 172.21.15.49:0/1463311552 shutdown_connections 2023-11-30T21:47:07.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:07.722+0000 7f231ba36700 1 -- 172.21.15.49:0/1463311552 wait complete. 2023-11-30T21:47:07.725 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:08.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:08 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:07.722328+0000 mon.smithi027 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.49:0/1463311552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:09.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:09.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:09.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:09 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:07.485798+0000 mgr.smithi027.brrjcl (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:10.313 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.311+0000 7fefa2534700 1 -- 172.21.15.49:0/2913861354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef9c0f5410 msgr2=0x7fef9c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:10.313 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.311+0000 7fefa2534700 1 --2- 172.21.15.49:0/2913861354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef9c0f5410 0x7fef9c0f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fef8c009fb0 tx=0x7fef8c009b30).stop 2023-11-30T21:47:10.313 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.311+0000 7fefa2534700 1 -- 172.21.15.49:0/2913861354 shutdown_connections 2023-11-30T21:47:10.313 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.311+0000 7fefa2534700 1 --2- 172.21.15.49:0/2913861354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef9c0f5410 0x7fef9c0f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:10.313 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.311+0000 7fefa2534700 1 -- 172.21.15.49:0/2913861354 >> 172.21.15.49:0/2913861354 conn(0x7fef9c0f0b70 msgr2=0x7fef9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:10.314 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.311+0000 7fefa2534700 1 -- 172.21.15.49:0/2913861354 shutdown_connections 2023-11-30T21:47:10.314 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.311+0000 7fefa2534700 1 -- 172.21.15.49:0/2913861354 wait complete. 2023-11-30T21:47:10.314 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.311+0000 7fefa2534700 1 Processor -- start 2023-11-30T21:47:10.314 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.312+0000 7fefa2534700 1 -- start start 2023-11-30T21:47:10.314 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.312+0000 7fefa2534700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef9c0f5410 0x7fef9c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:10.314 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.312+0000 7fefa2534700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c106c10 con 0x7fef9c0f5410 2023-11-30T21:47:10.315 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.312+0000 7fef9bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef9c0f5410 0x7fef9c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:10.315 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.312+0000 7fef9bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef9c0f5410 0x7fef9c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:57678/0 (socket says 172.21.15.49:57678) 2023-11-30T21:47:10.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.312+0000 7fef9bfff700 1 -- 172.21.15.49:0/4196204114 learned_addr learned my addr 172.21.15.49:0/4196204114 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:10.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.312+0000 7fef9bfff700 1 -- 172.21.15.49:0/4196204114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c005040 con 0x7fef9c0f5410 2023-11-30T21:47:10.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.313+0000 7fef9bfff700 1 --2- 172.21.15.49:0/4196204114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef9c0f5410 0x7fef9c106720 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fef8c013510 tx=0x7fef8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:10.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.313+0000 7fef98ff9700 1 -- 172.21.15.49:0/4196204114 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef8c009320 con 0x7fef9c0f5410 2023-11-30T21:47:10.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.313+0000 7fefa2534700 1 -- 172.21.15.49:0/4196204114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c106e10 con 0x7fef9c0f5410 2023-11-30T21:47:10.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.313+0000 7fef98ff9700 1 -- 172.21.15.49:0/4196204114 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef8c013f00 con 0x7fef9c0f5410 2023-11-30T21:47:10.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.313+0000 7fef98ff9700 1 -- 172.21.15.49:0/4196204114 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef8c014d90 con 0x7fef9c0f5410 2023-11-30T21:47:10.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.313+0000 7fefa2534700 1 -- 172.21.15.49:0/4196204114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c1072a0 con 0x7fef9c0f5410 2023-11-30T21:47:10.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.314+0000 7fef98ff9700 1 -- 172.21.15.49:0/4196204114 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef8c014420 con 0x7fef9c0f5410 2023-11-30T21:47:10.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.314+0000 7fefa2534700 1 -- 172.21.15.49:0/4196204114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c045ad0 con 0x7fef9c0f5410 2023-11-30T21:47:10.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.314+0000 7fef98ff9700 1 --2- 172.21.15.49:0/4196204114 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fef8802ff60 0x7fef88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:10.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.314+0000 7fef98ff9700 1 -- 172.21.15.49:0/4196204114 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef8c040ff0 con 0x7fef9c0f5410 2023-11-30T21:47:10.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.314+0000 7fef9b7fe700 1 --2- 172.21.15.49:0/4196204114 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fef8802ff60 0x7fef88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:10.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.315+0000 7fef9b7fe700 1 --2- 172.21.15.49:0/4196204114 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fef8802ff60 0x7fef88032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fef840096a0 tx=0x7fef84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:10.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.319+0000 7fef98ff9700 1 -- 172.21.15.49:0/4196204114 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef8c0095f0 con 0x7fef9c0f5410 2023-11-30T21:47:10.538 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.536+0000 7fefa2534700 1 -- 172.21.15.49:0/4196204114 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef9c108150 con 0x7fef9c0f5410 2023-11-30T21:47:10.538 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.537+0000 7fef98ff9700 1 -- 172.21.15.49:0/4196204114 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fef8c01db70 con 0x7fef9c0f5410 2023-11-30T21:47:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:10.539 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:10.540 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.538+0000 7fefa2534700 1 -- 172.21.15.49:0/4196204114 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fef8802ff60 msgr2=0x7fef88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:10.541 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.538+0000 7fefa2534700 1 --2- 172.21.15.49:0/4196204114 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fef8802ff60 0x7fef88032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fef840096a0 tx=0x7fef84006b40).stop 2023-11-30T21:47:10.541 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.538+0000 7fefa2534700 1 -- 172.21.15.49:0/4196204114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef9c0f5410 msgr2=0x7fef9c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:10.541 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.538+0000 7fefa2534700 1 --2- 172.21.15.49:0/4196204114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef9c0f5410 0x7fef9c106720 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fef8c013510 tx=0x7fef8c00b040).stop 2023-11-30T21:47:10.541 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.538+0000 7fefa2534700 1 -- 172.21.15.49:0/4196204114 shutdown_connections 2023-11-30T21:47:10.541 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.538+0000 7fefa2534700 1 --2- 172.21.15.49:0/4196204114 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fef8802ff60 0x7fef88032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:10.541 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.538+0000 7fefa2534700 1 --2- 172.21.15.49:0/4196204114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef9c0f5410 0x7fef9c106720 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:10.541 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.538+0000 7fefa2534700 1 -- 172.21.15.49:0/4196204114 >> 172.21.15.49:0/4196204114 conn(0x7fef9c0f0b70 msgr2=0x7fef9c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:10.541 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.539+0000 7fefa2534700 1 -- 172.21.15.49:0/4196204114 shutdown_connections 2023-11-30T21:47:10.541 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:10.539+0000 7fefa2534700 1 -- 172.21.15.49:0/4196204114 wait complete. 2023-11-30T21:47:10.541 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:11.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:09.486163+0000 mgr.smithi027.brrjcl (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:11.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:10.538546+0000 mon.smithi027 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.49:0/4196204114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:11.946 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:11.948 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:13.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.186+0000 7f6345461700 1 -- 172.21.15.49:0/1852925764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6340060280 msgr2=0x7f63400f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:13.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.186+0000 7f6345461700 1 --2- 172.21.15.49:0/1852925764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6340060280 0x7f63400f90f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6330009fb0 tx=0x7f6330009b30).stop 2023-11-30T21:47:13.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.186+0000 7f6345461700 1 -- 172.21.15.49:0/1852925764 shutdown_connections 2023-11-30T21:47:13.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.186+0000 7f6345461700 1 --2- 172.21.15.49:0/1852925764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6340060280 0x7f63400f90f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:13.189 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.186+0000 7f6345461700 1 -- 172.21.15.49:0/1852925764 >> 172.21.15.49:0/1852925764 conn(0x7f63400f0b70 msgr2=0x7f63400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:13.189 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.186+0000 7f6345461700 1 -- 172.21.15.49:0/1852925764 shutdown_connections 2023-11-30T21:47:13.189 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.187+0000 7f6345461700 1 -- 172.21.15.49:0/1852925764 wait complete. 2023-11-30T21:47:13.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.187+0000 7f6345461700 1 Processor -- start 2023-11-30T21:47:13.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.187+0000 7f6345461700 1 -- start start 2023-11-30T21:47:13.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.187+0000 7f6345461700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6340060280 0x7f63401066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.187+0000 7f6345461700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6340106bb0 con 0x7f6340060280 2023-11-30T21:47:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.188+0000 7f633effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6340060280 0x7f63401066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.188+0000 7f633effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6340060280 0x7f63401066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:57696/0 (socket says 172.21.15.49:57696) 2023-11-30T21:47:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.188+0000 7f633effd700 1 -- 172.21.15.49:0/3385761948 learned_addr learned my addr 172.21.15.49:0/3385761948 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.188+0000 7f633effd700 1 -- 172.21.15.49:0/3385761948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6330005040 con 0x7f6340060280 2023-11-30T21:47:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.188+0000 7f633effd700 1 --2- 172.21.15.49:0/3385761948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6340060280 0x7f63401066c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6330013510 tx=0x7f633000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.188+0000 7f63277fe700 1 -- 172.21.15.49:0/3385761948 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6330009320 con 0x7f6340060280 2023-11-30T21:47:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.188+0000 7f63277fe700 1 -- 172.21.15.49:0/3385761948 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6330013f00 con 0x7f6340060280 2023-11-30T21:47:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.188+0000 7f6345461700 1 -- 172.21.15.49:0/3385761948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6340106db0 con 0x7f6340060280 2023-11-30T21:47:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.188+0000 7f63277fe700 1 -- 172.21.15.49:0/3385761948 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6330014dc0 con 0x7f6340060280 2023-11-30T21:47:13.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.188+0000 7f6345461700 1 -- 172.21.15.49:0/3385761948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6340107240 con 0x7f6340060280 2023-11-30T21:47:13.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.189+0000 7f63277fe700 1 -- 172.21.15.49:0/3385761948 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63300095f0 con 0x7f6340060280 2023-11-30T21:47:13.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.189+0000 7f6345461700 1 -- 172.21.15.49:0/3385761948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6340045ad0 con 0x7f6340060280 2023-11-30T21:47:13.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.189+0000 7f63277fe700 1 --2- 172.21.15.49:0/3385761948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f632802ffb0 0x7f6328032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:13.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.189+0000 7f63277fe700 1 -- 172.21.15.49:0/3385761948 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6330040ff0 con 0x7f6340060280 2023-11-30T21:47:13.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.190+0000 7f633e7fc700 1 --2- 172.21.15.49:0/3385761948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f632802ffb0 0x7f6328032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:13.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.190+0000 7f633e7fc700 1 --2- 172.21.15.49:0/3385761948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f632802ffb0 0x7f6328032470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f63340096a0 tx=0x7f6334006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:13.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.194+0000 7f63277fe700 1 -- 172.21.15.49:0/3385761948 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f633006d2b0 con 0x7f6340060280 2023-11-30T21:47:13.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.415+0000 7f6345461700 1 -- 172.21.15.49:0/3385761948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63401080f0 con 0x7f6340060280 2023-11-30T21:47:13.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.416+0000 7f63277fe700 1 -- 172.21.15.49:0/3385761948 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6330014420 con 0x7f6340060280 2023-11-30T21:47:13.419 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:13.419 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:13.419 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.417+0000 7f6345461700 1 -- 172.21.15.49:0/3385761948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f632802ffb0 msgr2=0x7f6328032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:13.419 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.417+0000 7f6345461700 1 --2- 172.21.15.49:0/3385761948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f632802ffb0 0x7f6328032470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f63340096a0 tx=0x7f6334006b40).stop 2023-11-30T21:47:13.419 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.417+0000 7f6345461700 1 -- 172.21.15.49:0/3385761948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6340060280 msgr2=0x7f63401066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:13.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.417+0000 7f6345461700 1 --2- 172.21.15.49:0/3385761948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6340060280 0x7f63401066c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f6330013510 tx=0x7f633000b040).stop 2023-11-30T21:47:13.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.417+0000 7f6345461700 1 -- 172.21.15.49:0/3385761948 shutdown_connections 2023-11-30T21:47:13.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.417+0000 7f6345461700 1 --2- 172.21.15.49:0/3385761948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f632802ffb0 0x7f6328032470 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:13.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.417+0000 7f6345461700 1 --2- 172.21.15.49:0/3385761948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6340060280 0x7f63401066c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:13.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.417+0000 7f6345461700 1 -- 172.21.15.49:0/3385761948 >> 172.21.15.49:0/3385761948 conn(0x7f63400f0b70 msgr2=0x7f63400f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:13.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.417+0000 7f6345461700 1 -- 172.21.15.49:0/3385761948 shutdown_connections 2023-11-30T21:47:13.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:13.417+0000 7f6345461700 1 -- 172.21.15.49:0/3385761948 wait complete. 2023-11-30T21:47:13.420 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:13.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:13 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:11.486598+0000 mgr.smithi027.brrjcl (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:14.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:13.417221+0000 mon.smithi027 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.49:0/3385761948' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:14.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:14.791 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:15.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:15 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:13.486933+0000 mgr.smithi027.brrjcl (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:15.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.985+0000 7f0935ce7700 1 -- 172.21.15.49:0/112771480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0930060280 msgr2=0x7f09300f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:15.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.985+0000 7f0935ce7700 1 --2- 172.21.15.49:0/112771480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0930060280 0x7f09300f90f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f0920009fb0 tx=0x7f0920009b30).stop 2023-11-30T21:47:15.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.985+0000 7f0935ce7700 1 -- 172.21.15.49:0/112771480 shutdown_connections 2023-11-30T21:47:15.988 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.985+0000 7f0935ce7700 1 --2- 172.21.15.49:0/112771480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0930060280 0x7f09300f90f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.985+0000 7f0935ce7700 1 -- 172.21.15.49:0/112771480 >> 172.21.15.49:0/112771480 conn(0x7f09300f0b70 msgr2=0x7f09300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.985+0000 7f0935ce7700 1 -- 172.21.15.49:0/112771480 shutdown_connections 2023-11-30T21:47:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.985+0000 7f0935ce7700 1 -- 172.21.15.49:0/112771480 wait complete. 2023-11-30T21:47:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.986+0000 7f0935ce7700 1 Processor -- start 2023-11-30T21:47:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.986+0000 7f0935ce7700 1 -- start start 2023-11-30T21:47:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.986+0000 7f0935ce7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0930060280 0x7f09301066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.986+0000 7f0935ce7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0930106bd0 con 0x7f0930060280 2023-11-30T21:47:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.987+0000 7f092f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0930060280 0x7f09301066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.987+0000 7f092f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0930060280 0x7f09301066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:57722/0 (socket says 172.21.15.49:57722) 2023-11-30T21:47:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.987+0000 7f092f7fe700 1 -- 172.21.15.49:0/4131237838 learned_addr learned my addr 172.21.15.49:0/4131237838 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.987+0000 7f092f7fe700 1 -- 172.21.15.49:0/4131237838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0920005040 con 0x7f0930060280 2023-11-30T21:47:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.987+0000 7f092f7fe700 1 --2- 172.21.15.49:0/4131237838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0930060280 0x7f09301066e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f0920013510 tx=0x7f092000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:15.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.987+0000 7f0927fff700 1 -- 172.21.15.49:0/4131237838 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0920009320 con 0x7f0930060280 2023-11-30T21:47:15.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.987+0000 7f0935ce7700 1 -- 172.21.15.49:0/4131237838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0930106dd0 con 0x7f0930060280 2023-11-30T21:47:15.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.987+0000 7f0927fff700 1 -- 172.21.15.49:0/4131237838 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0920013f00 con 0x7f0930060280 2023-11-30T21:47:15.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.987+0000 7f0927fff700 1 -- 172.21.15.49:0/4131237838 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0920014dc0 con 0x7f0930060280 2023-11-30T21:47:15.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.987+0000 7f0935ce7700 1 -- 172.21.15.49:0/4131237838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0930107260 con 0x7f0930060280 2023-11-30T21:47:15.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.988+0000 7f0927fff700 1 -- 172.21.15.49:0/4131237838 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f092001d3d0 con 0x7f0930060280 2023-11-30T21:47:15.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.988+0000 7f0935ce7700 1 -- 172.21.15.49:0/4131237838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0930045ad0 con 0x7f0930060280 2023-11-30T21:47:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.988+0000 7f0927fff700 1 --2- 172.21.15.49:0/4131237838 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f091c02ff60 0x7f091c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.988+0000 7f0927fff700 1 -- 172.21.15.49:0/4131237838 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09200410d0 con 0x7f0930060280 2023-11-30T21:47:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.988+0000 7f092effd700 1 --2- 172.21.15.49:0/4131237838 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f091c02ff60 0x7f091c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.989+0000 7f092effd700 1 --2- 172.21.15.49:0/4131237838 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f091c02ff60 0x7f091c032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f09180096a0 tx=0x7f0918006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:15.993+0000 7f0927fff700 1 -- 172.21.15.49:0/4131237838 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09200126f0 con 0x7f0930060280 2023-11-30T21:47:16.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:16.210+0000 7f0935ce7700 1 -- 172.21.15.49:0/4131237838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0930108110 con 0x7f0930060280 2023-11-30T21:47:16.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:16.211+0000 7f0927fff700 1 -- 172.21.15.49:0/4131237838 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f092000f340 con 0x7f0930060280 2023-11-30T21:47:16.212 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:16.213 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:16.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:16.212+0000 7f0935ce7700 1 -- 172.21.15.49:0/4131237838 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f091c02ff60 msgr2=0x7f091c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:16.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:16.212+0000 7f0935ce7700 1 --2- 172.21.15.49:0/4131237838 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f091c02ff60 0x7f091c032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f09180096a0 tx=0x7f0918006b40).stop 2023-11-30T21:47:16.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:16.212+0000 7f0935ce7700 1 -- 172.21.15.49:0/4131237838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0930060280 msgr2=0x7f09301066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:16.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:16.212+0000 7f0935ce7700 1 --2- 172.21.15.49:0/4131237838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0930060280 0x7f09301066e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f0920013510 tx=0x7f092000b040).stop 2023-11-30T21:47:16.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:16.212+0000 7f0935ce7700 1 -- 172.21.15.49:0/4131237838 shutdown_connections 2023-11-30T21:47:16.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:16.212+0000 7f0935ce7700 1 --2- 172.21.15.49:0/4131237838 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f091c02ff60 0x7f091c032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:16.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:16.212+0000 7f0935ce7700 1 --2- 172.21.15.49:0/4131237838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0930060280 0x7f09301066e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:16.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:16.212+0000 7f0935ce7700 1 -- 172.21.15.49:0/4131237838 >> 172.21.15.49:0/4131237838 conn(0x7f09300f0b70 msgr2=0x7f09300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:16.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:16.212+0000 7f0935ce7700 1 -- 172.21.15.49:0/4131237838 shutdown_connections 2023-11-30T21:47:16.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:16.212+0000 7f0935ce7700 1 -- 172.21.15.49:0/4131237838 wait complete. 2023-11-30T21:47:16.215 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:17.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:15.487245+0000 mgr.smithi027.brrjcl (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:17.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:16.212342+0000 mon.smithi027 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.49:0/4131237838' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:17.589 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:17.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:18.758 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.757+0000 7f7847fff700 1 -- 172.21.15.49:0/1030536237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78480601e0 msgr2=0x7f78480f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:18.758 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.757+0000 7f7847fff700 1 --2- 172.21.15.49:0/1030536237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78480601e0 0x7f78480f8f20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f7838009fb0 tx=0x7f7838009b30).stop 2023-11-30T21:47:18.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.757+0000 7f7847fff700 1 -- 172.21.15.49:0/1030536237 shutdown_connections 2023-11-30T21:47:18.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.757+0000 7f7847fff700 1 --2- 172.21.15.49:0/1030536237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78480601e0 0x7f78480f8f20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:18.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.757+0000 7f7847fff700 1 -- 172.21.15.49:0/1030536237 >> 172.21.15.49:0/1030536237 conn(0x7f78480f09a0 msgr2=0x7f78480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:18.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.757+0000 7f7847fff700 1 -- 172.21.15.49:0/1030536237 shutdown_connections 2023-11-30T21:47:18.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.757+0000 7f7847fff700 1 -- 172.21.15.49:0/1030536237 wait complete. 2023-11-30T21:47:18.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.757+0000 7f7847fff700 1 Processor -- start 2023-11-30T21:47:18.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.757+0000 7f7847fff700 1 -- start start 2023-11-30T21:47:18.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.758+0000 7f7847fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78480601e0 0x7f7848106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:18.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.758+0000 7f7847fff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78481069f0 con 0x7f78480601e0 2023-11-30T21:47:18.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.758+0000 7f7846ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78480601e0 0x7f7848106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:18.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.758+0000 7f7846ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78480601e0 0x7f7848106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:55088/0 (socket says 172.21.15.49:55088) 2023-11-30T21:47:18.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.758+0000 7f7846ffd700 1 -- 172.21.15.49:0/3894606803 learned_addr learned my addr 172.21.15.49:0/3894606803 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:18.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.758+0000 7f7846ffd700 1 -- 172.21.15.49:0/3894606803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7838005040 con 0x7f78480601e0 2023-11-30T21:47:18.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.759+0000 7f7846ffd700 1 --2- 172.21.15.49:0/3894606803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78480601e0 0x7f7848106500 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f7838013510 tx=0x7f783800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:18.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.759+0000 7f782f7fe700 1 -- 172.21.15.49:0/3894606803 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7838009320 con 0x7f78480601e0 2023-11-30T21:47:18.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.759+0000 7f782f7fe700 1 -- 172.21.15.49:0/3894606803 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7838013f00 con 0x7f78480601e0 2023-11-30T21:47:18.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.759+0000 7f7847fff700 1 -- 172.21.15.49:0/3894606803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7848106bf0 con 0x7f78480601e0 2023-11-30T21:47:18.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.759+0000 7f782f7fe700 1 -- 172.21.15.49:0/3894606803 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7838014d90 con 0x7f78480601e0 2023-11-30T21:47:18.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.759+0000 7f7847fff700 1 -- 172.21.15.49:0/3894606803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7848107080 con 0x7f78480601e0 2023-11-30T21:47:18.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.760+0000 7f782f7fe700 1 -- 172.21.15.49:0/3894606803 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78380095f0 con 0x7f78480601e0 2023-11-30T21:47:18.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.760+0000 7f7847fff700 1 -- 172.21.15.49:0/3894606803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7848045ad0 con 0x7f78480601e0 2023-11-30T21:47:18.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.760+0000 7f782f7fe700 1 --2- 172.21.15.49:0/3894606803 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f783002ff60 0x7f7830032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:18.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.760+0000 7f782f7fe700 1 -- 172.21.15.49:0/3894606803 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7838041000 con 0x7f78480601e0 2023-11-30T21:47:18.764 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.760+0000 7f78467fc700 1 --2- 172.21.15.49:0/3894606803 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f783002ff60 0x7f7830032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:18.764 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.761+0000 7f78467fc700 1 --2- 172.21.15.49:0/3894606803 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f783002ff60 0x7f7830032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f783c0096a0 tx=0x7f783c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:18.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.765+0000 7f782f7fe700 1 -- 172.21.15.49:0/3894606803 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7838014420 con 0x7f78480601e0 2023-11-30T21:47:18.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.987+0000 7f7847fff700 1 -- 172.21.15.49:0/3894606803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7848107f30 con 0x7f78480601e0 2023-11-30T21:47:18.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.988+0000 7f782f7fe700 1 -- 172.21.15.49:0/3894606803 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f783801d7f0 con 0x7f78480601e0 2023-11-30T21:47:18.991 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:18.991 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:18.991 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.990+0000 7f7847fff700 1 -- 172.21.15.49:0/3894606803 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f783002ff60 msgr2=0x7f7830032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:18.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.990+0000 7f7847fff700 1 --2- 172.21.15.49:0/3894606803 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f783002ff60 0x7f7830032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f783c0096a0 tx=0x7f783c006b40).stop 2023-11-30T21:47:18.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.990+0000 7f7847fff700 1 -- 172.21.15.49:0/3894606803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78480601e0 msgr2=0x7f7848106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:18.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.990+0000 7f7847fff700 1 --2- 172.21.15.49:0/3894606803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78480601e0 0x7f7848106500 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f7838013510 tx=0x7f783800b040).stop 2023-11-30T21:47:18.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.990+0000 7f7847fff700 1 -- 172.21.15.49:0/3894606803 shutdown_connections 2023-11-30T21:47:18.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.990+0000 7f7847fff700 1 --2- 172.21.15.49:0/3894606803 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f783002ff60 0x7f7830032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:18.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.990+0000 7f7847fff700 1 --2- 172.21.15.49:0/3894606803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78480601e0 0x7f7848106500 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:18.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.990+0000 7f7847fff700 1 -- 172.21.15.49:0/3894606803 >> 172.21.15.49:0/3894606803 conn(0x7f78480f09a0 msgr2=0x7f78480f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:18.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.990+0000 7f7847fff700 1 -- 172.21.15.49:0/3894606803 shutdown_connections 2023-11-30T21:47:18.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:18.990+0000 7f7847fff700 1 -- 172.21.15.49:0/3894606803 wait complete. 2023-11-30T21:47:18.992 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:19.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:17.487546+0000 mgr.smithi027.brrjcl (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:19.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:18.989369+0000 mon.smithi027 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.49:0/3894606803' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:20.403 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:20.404 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:21.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:21 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:19.487885+0000 mgr.smithi027.brrjcl (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:21.639 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.633+0000 7f4097a54700 1 -- 172.21.15.49:0/2037881583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40900f6510 msgr2=0x7f40900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:21.639 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.633+0000 7f4097a54700 1 --2- 172.21.15.49:0/2037881583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40900f6510 0x7f40900f6930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f408c009fb0 tx=0x7f408c009b30).stop 2023-11-30T21:47:21.639 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.633+0000 7f4097a54700 1 -- 172.21.15.49:0/2037881583 shutdown_connections 2023-11-30T21:47:21.639 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.633+0000 7f4097a54700 1 --2- 172.21.15.49:0/2037881583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40900f6510 0x7f40900f6930 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:21.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.633+0000 7f4097a54700 1 -- 172.21.15.49:0/2037881583 >> 172.21.15.49:0/2037881583 conn(0x7f40900f1c70 msgr2=0x7f40900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:21.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.634+0000 7f4097a54700 1 -- 172.21.15.49:0/2037881583 shutdown_connections 2023-11-30T21:47:21.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.634+0000 7f4097a54700 1 -- 172.21.15.49:0/2037881583 wait complete. 2023-11-30T21:47:21.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.634+0000 7f4097a54700 1 Processor -- start 2023-11-30T21:47:21.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.634+0000 7f4097a54700 1 -- start start 2023-11-30T21:47:21.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.634+0000 7f4097a54700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40900f6510 0x7f40901066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:21.641 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.634+0000 7f4097a54700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4090106bc0 con 0x7f40900f6510 2023-11-30T21:47:21.641 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.635+0000 7f40957f0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40900f6510 0x7f40901066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:21.641 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.635+0000 7f40957f0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40900f6510 0x7f40901066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:55106/0 (socket says 172.21.15.49:55106) 2023-11-30T21:47:21.641 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.635+0000 7f40957f0700 1 -- 172.21.15.49:0/1731689591 learned_addr learned my addr 172.21.15.49:0/1731689591 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:21.641 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.635+0000 7f40957f0700 1 -- 172.21.15.49:0/1731689591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c005040 con 0x7f40900f6510 2023-11-30T21:47:21.642 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.635+0000 7f40957f0700 1 --2- 172.21.15.49:0/1731689591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40900f6510 0x7f40901066d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f408c013510 tx=0x7f408c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:21.642 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.635+0000 7f4081ffb700 1 -- 172.21.15.49:0/1731689591 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f408c009320 con 0x7f40900f6510 2023-11-30T21:47:21.642 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.635+0000 7f4097a54700 1 -- 172.21.15.49:0/1731689591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4090106dc0 con 0x7f40900f6510 2023-11-30T21:47:21.642 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.635+0000 7f4081ffb700 1 -- 172.21.15.49:0/1731689591 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f408c013f00 con 0x7f40900f6510 2023-11-30T21:47:21.642 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.635+0000 7f4081ffb700 1 -- 172.21.15.49:0/1731689591 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f408c014d90 con 0x7f40900f6510 2023-11-30T21:47:21.642 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.635+0000 7f4097a54700 1 -- 172.21.15.49:0/1731689591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4090107250 con 0x7f40900f6510 2023-11-30T21:47:21.642 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.636+0000 7f4081ffb700 1 -- 172.21.15.49:0/1731689591 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f408c0095f0 con 0x7f40900f6510 2023-11-30T21:47:21.642 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.636+0000 7f4097a54700 1 -- 172.21.15.49:0/1731689591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4090045ad0 con 0x7f40900f6510 2023-11-30T21:47:21.642 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.636+0000 7f4081ffb700 1 --2- 172.21.15.49:0/1731689591 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f407c02ffb0 0x7f407c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:21.643 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.636+0000 7f4081ffb700 1 -- 172.21.15.49:0/1731689591 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f408c0410c0 con 0x7f40900f6510 2023-11-30T21:47:21.643 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.637+0000 7f4094fef700 1 --2- 172.21.15.49:0/1731689591 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f407c02ffb0 0x7f407c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:21.643 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.637+0000 7f4094fef700 1 --2- 172.21.15.49:0/1731689591 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f407c02ffb0 0x7f407c032470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f40840096a0 tx=0x7f4084006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:21.643 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.641+0000 7f4081ffb700 1 -- 172.21.15.49:0/1731689591 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f408c01d7f0 con 0x7f40900f6510 2023-11-30T21:47:21.866 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.864+0000 7f4097a54700 1 -- 172.21.15.49:0/1731689591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4090108100 con 0x7f40900f6510 2023-11-30T21:47:21.866 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.865+0000 7f4081ffb700 1 -- 172.21.15.49:0/1731689591 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f408c010230 con 0x7f40900f6510 2023-11-30T21:47:21.867 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:21.867 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:21.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.867+0000 7f4097a54700 1 -- 172.21.15.49:0/1731689591 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f407c02ffb0 msgr2=0x7f407c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:21.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.867+0000 7f4097a54700 1 --2- 172.21.15.49:0/1731689591 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f407c02ffb0 0x7f407c032470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f40840096a0 tx=0x7f4084006b40).stop 2023-11-30T21:47:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.867+0000 7f4097a54700 1 -- 172.21.15.49:0/1731689591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40900f6510 msgr2=0x7f40901066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.867+0000 7f4097a54700 1 --2- 172.21.15.49:0/1731689591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40900f6510 0x7f40901066d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f408c013510 tx=0x7f408c00b040).stop 2023-11-30T21:47:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.867+0000 7f4097a54700 1 -- 172.21.15.49:0/1731689591 shutdown_connections 2023-11-30T21:47:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.867+0000 7f4097a54700 1 --2- 172.21.15.49:0/1731689591 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f407c02ffb0 0x7f407c032470 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.867+0000 7f4097a54700 1 --2- 172.21.15.49:0/1731689591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40900f6510 0x7f40901066d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.867+0000 7f4097a54700 1 -- 172.21.15.49:0/1731689591 >> 172.21.15.49:0/1731689591 conn(0x7f40900f1c70 msgr2=0x7f40900f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.867+0000 7f4097a54700 1 -- 172.21.15.49:0/1731689591 shutdown_connections 2023-11-30T21:47:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:21.867+0000 7f4097a54700 1 -- 172.21.15.49:0/1731689591 wait complete. 2023-11-30T21:47:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:22.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:22 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:21.866542+0000 mon.smithi027 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.49:0/1731689591' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:23.274 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:23.274 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:23.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:23 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:21.488234+0000 mgr.smithi027.brrjcl (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:24.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.516+0000 7fc0b0629700 1 -- 172.21.15.49:0/16227623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a80f6510 msgr2=0x7fc0a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:24.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.516+0000 7fc0b0629700 1 --2- 172.21.15.49:0/16227623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a80f6510 0x7fc0a80f6930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fc0a4009fb0 tx=0x7fc0a4009b30).stop 2023-11-30T21:47:24.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.516+0000 7fc0b0629700 1 -- 172.21.15.49:0/16227623 shutdown_connections 2023-11-30T21:47:24.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.516+0000 7fc0b0629700 1 --2- 172.21.15.49:0/16227623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a80f6510 0x7fc0a80f6930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:24.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.516+0000 7fc0b0629700 1 -- 172.21.15.49:0/16227623 >> 172.21.15.49:0/16227623 conn(0x7fc0a80f1c70 msgr2=0x7fc0a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:24.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.516+0000 7fc0b0629700 1 -- 172.21.15.49:0/16227623 shutdown_connections 2023-11-30T21:47:24.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.516+0000 7fc0b0629700 1 -- 172.21.15.49:0/16227623 wait complete. 2023-11-30T21:47:24.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.517+0000 7fc0b0629700 1 Processor -- start 2023-11-30T21:47:24.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.517+0000 7fc0b0629700 1 -- start start 2023-11-30T21:47:24.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.517+0000 7fc0b0629700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a80f6510 0x7fc0a8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:24.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.517+0000 7fc0b0629700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a8106c20 con 0x7fc0a80f6510 2023-11-30T21:47:24.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.517+0000 7fc0ae3c5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a80f6510 0x7fc0a8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:24.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.517+0000 7fc0ae3c5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a80f6510 0x7fc0a8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:55130/0 (socket says 172.21.15.49:55130) 2023-11-30T21:47:24.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.517+0000 7fc0ae3c5700 1 -- 172.21.15.49:0/173025000 learned_addr learned my addr 172.21.15.49:0/173025000 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:24.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.517+0000 7fc0ae3c5700 1 -- 172.21.15.49:0/173025000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a4005040 con 0x7fc0a80f6510 2023-11-30T21:47:24.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.518+0000 7fc0ae3c5700 1 --2- 172.21.15.49:0/173025000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a80f6510 0x7fc0a8106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fc0a4013510 tx=0x7fc0a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:24.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.518+0000 7fc09effd700 1 -- 172.21.15.49:0/173025000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0a4009320 con 0x7fc0a80f6510 2023-11-30T21:47:24.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.518+0000 7fc0b0629700 1 -- 172.21.15.49:0/173025000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a8106e20 con 0x7fc0a80f6510 2023-11-30T21:47:24.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.518+0000 7fc09effd700 1 -- 172.21.15.49:0/173025000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0a4013f00 con 0x7fc0a80f6510 2023-11-30T21:47:24.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.518+0000 7fc09effd700 1 -- 172.21.15.49:0/173025000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0a4014d90 con 0x7fc0a80f6510 2023-11-30T21:47:24.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.518+0000 7fc0b0629700 1 -- 172.21.15.49:0/173025000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a8107230 con 0x7fc0a80f6510 2023-11-30T21:47:24.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.519+0000 7fc09effd700 1 -- 172.21.15.49:0/173025000 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0a40095f0 con 0x7fc0a80f6510 2023-11-30T21:47:24.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.519+0000 7fc0b0629700 1 -- 172.21.15.49:0/173025000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a8045ad0 con 0x7fc0a80f6510 2023-11-30T21:47:24.523 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.519+0000 7fc09effd700 1 --2- 172.21.15.49:0/173025000 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc094034370 0x7fc094036830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:24.523 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.519+0000 7fc09effd700 1 -- 172.21.15.49:0/173025000 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0a4041080 con 0x7fc0a80f6510 2023-11-30T21:47:24.523 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.519+0000 7fc0adbc4700 1 --2- 172.21.15.49:0/173025000 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc094034370 0x7fc094036830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:24.523 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.520+0000 7fc0adbc4700 1 --2- 172.21.15.49:0/173025000 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc094034370 0x7fc094036830 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fc0980096a0 tx=0x7fc098006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:24.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.524+0000 7fc09effd700 1 -- 172.21.15.49:0/173025000 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0a4014420 con 0x7fc0a80f6510 2023-11-30T21:47:24.744 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.743+0000 7fc0b0629700 1 -- 172.21.15.49:0/173025000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0a80fbf60 con 0x7fc0a80f6510 2023-11-30T21:47:24.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.743+0000 7fc09effd700 1 -- 172.21.15.49:0/173025000 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc0a4010230 con 0x7fc0a80f6510 2023-11-30T21:47:24.746 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:24.746 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:24.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.745+0000 7fc0b0629700 1 -- 172.21.15.49:0/173025000 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc094034370 msgr2=0x7fc094036830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:24.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.745+0000 7fc0b0629700 1 --2- 172.21.15.49:0/173025000 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc094034370 0x7fc094036830 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fc0980096a0 tx=0x7fc098006b40).stop 2023-11-30T21:47:24.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.745+0000 7fc0b0629700 1 -- 172.21.15.49:0/173025000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a80f6510 msgr2=0x7fc0a8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:24.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.745+0000 7fc0b0629700 1 --2- 172.21.15.49:0/173025000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a80f6510 0x7fc0a8106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fc0a4013510 tx=0x7fc0a400b040).stop 2023-11-30T21:47:24.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.745+0000 7fc0b0629700 1 -- 172.21.15.49:0/173025000 shutdown_connections 2023-11-30T21:47:24.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.745+0000 7fc0b0629700 1 --2- 172.21.15.49:0/173025000 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc094034370 0x7fc094036830 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:24.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.745+0000 7fc0b0629700 1 --2- 172.21.15.49:0/173025000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0a80f6510 0x7fc0a8106730 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:24.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.745+0000 7fc0b0629700 1 -- 172.21.15.49:0/173025000 >> 172.21.15.49:0/173025000 conn(0x7fc0a80f1c70 msgr2=0x7fc0a80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:24.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.745+0000 7fc0b0629700 1 -- 172.21.15.49:0/173025000 shutdown_connections 2023-11-30T21:47:24.748 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:24.745+0000 7fc0b0629700 1 -- 172.21.15.49:0/173025000 wait complete. 2023-11-30T21:47:24.748 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:25.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:23.488523+0000 mgr.smithi027.brrjcl (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:25.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:24.744812+0000 mon.smithi027 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.49:0/173025000' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:26.135 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:26.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:27.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.403+0000 7f799f59e700 1 -- 172.21.15.49:0/1156112691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a00f51a0 msgr2=0x7f79a00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:27.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.403+0000 7f799f59e700 1 --2- 172.21.15.49:0/1156112691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a00f51a0 0x7f79a00f55c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f7990009fb0 tx=0x7f7990009b30).stop 2023-11-30T21:47:27.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.403+0000 7f799f59e700 1 -- 172.21.15.49:0/1156112691 shutdown_connections 2023-11-30T21:47:27.406 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.403+0000 7f799f59e700 1 --2- 172.21.15.49:0/1156112691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a00f51a0 0x7f79a00f55c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:27.406 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.403+0000 7f799f59e700 1 -- 172.21.15.49:0/1156112691 >> 172.21.15.49:0/1156112691 conn(0x7f79a00f09a0 msgr2=0x7f79a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:27.406 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.403+0000 7f799f59e700 1 -- 172.21.15.49:0/1156112691 shutdown_connections 2023-11-30T21:47:27.406 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.403+0000 7f799f59e700 1 -- 172.21.15.49:0/1156112691 wait complete. 2023-11-30T21:47:27.406 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.404+0000 7f799f59e700 1 Processor -- start 2023-11-30T21:47:27.406 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.404+0000 7f799f59e700 1 -- start start 2023-11-30T21:47:27.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.404+0000 7f799f59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a00f51a0 0x7f79a01091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:27.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.404+0000 7f799f59e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a0060220 con 0x7f79a00f51a0 2023-11-30T21:47:27.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.404+0000 7f799e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a00f51a0 0x7f79a01091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:27.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.404+0000 7f799e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a00f51a0 0x7f79a01091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:55142/0 (socket says 172.21.15.49:55142) 2023-11-30T21:47:27.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.404+0000 7f799e59c700 1 -- 172.21.15.49:0/3910070957 learned_addr learned my addr 172.21.15.49:0/3910070957 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:27.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.405+0000 7f799e59c700 1 -- 172.21.15.49:0/3910070957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7990005040 con 0x7f79a00f51a0 2023-11-30T21:47:27.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.405+0000 7f799e59c700 1 --2- 172.21.15.49:0/3910070957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a00f51a0 0x7f79a01091d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f7990009830 tx=0x7f799000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:27.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.405+0000 7f798effd700 1 -- 172.21.15.49:0/3910070957 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7990009320 con 0x7f79a00f51a0 2023-11-30T21:47:27.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.405+0000 7f799f59e700 1 -- 172.21.15.49:0/3910070957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a0060420 con 0x7f79a00f51a0 2023-11-30T21:47:27.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.405+0000 7f798effd700 1 -- 172.21.15.49:0/3910070957 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79900136c0 con 0x7f79a00f51a0 2023-11-30T21:47:27.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.405+0000 7f798effd700 1 -- 172.21.15.49:0/3910070957 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f799001ee90 con 0x7f79a00f51a0 2023-11-30T21:47:27.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.405+0000 7f799f59e700 1 -- 172.21.15.49:0/3910070957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a0109880 con 0x7f79a00f51a0 2023-11-30T21:47:27.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.406+0000 7f798effd700 1 -- 172.21.15.49:0/3910070957 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f799001e420 con 0x7f79a00f51a0 2023-11-30T21:47:27.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.406+0000 7f799f59e700 1 -- 172.21.15.49:0/3910070957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a0059100 con 0x7f79a00f51a0 2023-11-30T21:47:27.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.406+0000 7f798effd700 1 --2- 172.21.15.49:0/3910070957 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f798802ff10 0x7f79880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:27.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.406+0000 7f798effd700 1 -- 172.21.15.49:0/3910070957 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7990041050 con 0x7f79a00f51a0 2023-11-30T21:47:27.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.406+0000 7f799dd9b700 1 --2- 172.21.15.49:0/3910070957 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f798802ff10 0x7f79880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:27.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.407+0000 7f799dd9b700 1 --2- 172.21.15.49:0/3910070957 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f798802ff10 0x7f79880323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f79940096a0 tx=0x7f7994006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:27.412 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.411+0000 7f798effd700 1 -- 172.21.15.49:0/3910070957 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f799006d2b0 con 0x7f79a00f51a0 2023-11-30T21:47:27.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:27 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:25.488827+0000 mgr.smithi027.brrjcl (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:27.635 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.631+0000 7f799f59e700 1 -- 172.21.15.49:0/3910070957 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79a0045ad0 con 0x7f79a00f51a0 2023-11-30T21:47:27.635 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.632+0000 7f798effd700 1 -- 172.21.15.49:0/3910070957 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f79900095f0 con 0x7f79a00f51a0 2023-11-30T21:47:27.635 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:27.635 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:27.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.634+0000 7f799f59e700 1 -- 172.21.15.49:0/3910070957 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f798802ff10 msgr2=0x7f79880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:27.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.634+0000 7f799f59e700 1 --2- 172.21.15.49:0/3910070957 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f798802ff10 0x7f79880323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f79940096a0 tx=0x7f7994006b40).stop 2023-11-30T21:47:27.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.634+0000 7f799f59e700 1 -- 172.21.15.49:0/3910070957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a00f51a0 msgr2=0x7f79a01091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:27.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.634+0000 7f799f59e700 1 --2- 172.21.15.49:0/3910070957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a00f51a0 0x7f79a01091d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f7990009830 tx=0x7f799000b040).stop 2023-11-30T21:47:27.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.634+0000 7f799f59e700 1 -- 172.21.15.49:0/3910070957 shutdown_connections 2023-11-30T21:47:27.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.634+0000 7f799f59e700 1 --2- 172.21.15.49:0/3910070957 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f798802ff10 0x7f79880323d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:27.638 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.634+0000 7f799f59e700 1 --2- 172.21.15.49:0/3910070957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79a00f51a0 0x7f79a01091d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:27.638 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.634+0000 7f799f59e700 1 -- 172.21.15.49:0/3910070957 >> 172.21.15.49:0/3910070957 conn(0x7f79a00f09a0 msgr2=0x7f79a00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:27.638 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.634+0000 7f799f59e700 1 -- 172.21.15.49:0/3910070957 shutdown_connections 2023-11-30T21:47:27.638 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:27.634+0000 7f799f59e700 1 -- 172.21.15.49:0/3910070957 wait complete. 2023-11-30T21:47:27.638 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:28.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:28 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:27.633682+0000 mon.smithi027 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.49:0/3910070957' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:29.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:29.026 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:29.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:27.489126+0000 mgr.smithi027.brrjcl (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.185+0000 7f14c68e4700 1 -- 172.21.15.49:0/2248491654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c00f5410 msgr2=0x7f14c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.185+0000 7f14c68e4700 1 --2- 172.21.15.49:0/2248491654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c00f5410 0x7f14c00f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f14b0009fb0 tx=0x7f14b0009b30).stop 2023-11-30T21:47:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.185+0000 7f14c68e4700 1 -- 172.21.15.49:0/2248491654 shutdown_connections 2023-11-30T21:47:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.185+0000 7f14c68e4700 1 --2- 172.21.15.49:0/2248491654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c00f5410 0x7f14c00f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.185+0000 7f14c68e4700 1 -- 172.21.15.49:0/2248491654 >> 172.21.15.49:0/2248491654 conn(0x7f14c00f0b70 msgr2=0x7f14c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.185+0000 7f14c68e4700 1 -- 172.21.15.49:0/2248491654 shutdown_connections 2023-11-30T21:47:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.186+0000 7f14c68e4700 1 -- 172.21.15.49:0/2248491654 wait complete. 2023-11-30T21:47:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.186+0000 7f14c68e4700 1 Processor -- start 2023-11-30T21:47:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.186+0000 7f14c68e4700 1 -- start start 2023-11-30T21:47:30.189 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.186+0000 7f14c68e4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c00f5410 0x7f14c0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:30.189 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.186+0000 7f14c68e4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c0060220 con 0x7f14c00f5410 2023-11-30T21:47:30.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.187+0000 7f14bffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c00f5410 0x7f14c0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:30.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.187+0000 7f14bffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c00f5410 0x7f14c0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:50870/0 (socket says 172.21.15.49:50870) 2023-11-30T21:47:30.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.187+0000 7f14bffff700 1 -- 172.21.15.49:0/531711237 learned_addr learned my addr 172.21.15.49:0/531711237 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:30.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.187+0000 7f14bffff700 1 -- 172.21.15.49:0/531711237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b0005040 con 0x7f14c00f5410 2023-11-30T21:47:30.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.187+0000 7f14bffff700 1 --2- 172.21.15.49:0/531711237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c00f5410 0x7f14c0109450 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f14b0013510 tx=0x7f14b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:30.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.187+0000 7f14bcff9700 1 -- 172.21.15.49:0/531711237 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14b0009320 con 0x7f14c00f5410 2023-11-30T21:47:30.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.187+0000 7f14c68e4700 1 -- 172.21.15.49:0/531711237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c0060420 con 0x7f14c00f5410 2023-11-30T21:47:30.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.187+0000 7f14bcff9700 1 -- 172.21.15.49:0/531711237 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14b0013f00 con 0x7f14c00f5410 2023-11-30T21:47:30.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.187+0000 7f14bcff9700 1 -- 172.21.15.49:0/531711237 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14b0014d90 con 0x7f14c00f5410 2023-11-30T21:47:30.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.187+0000 7f14c68e4700 1 -- 172.21.15.49:0/531711237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c0109b00 con 0x7f14c00f5410 2023-11-30T21:47:30.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.188+0000 7f14bcff9700 1 -- 172.21.15.49:0/531711237 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14b0014420 con 0x7f14c00f5410 2023-11-30T21:47:30.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.188+0000 7f14c68e4700 1 -- 172.21.15.49:0/531711237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c0059100 con 0x7f14c00f5410 2023-11-30T21:47:30.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.188+0000 7f14bcff9700 1 --2- 172.21.15.49:0/531711237 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f14a802ff60 0x7f14a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:30.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.188+0000 7f14bcff9700 1 -- 172.21.15.49:0/531711237 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14b0041000 con 0x7f14c00f5410 2023-11-30T21:47:30.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.188+0000 7f14bf7fe700 1 --2- 172.21.15.49:0/531711237 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f14a802ff60 0x7f14a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:30.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.189+0000 7f14bf7fe700 1 --2- 172.21.15.49:0/531711237 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f14a802ff60 0x7f14a8032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f14b40096a0 tx=0x7f14b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:30.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.194+0000 7f14bcff9700 1 -- 172.21.15.49:0/531711237 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14b00095f0 con 0x7f14c00f5410 2023-11-30T21:47:30.413 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.413+0000 7f14c68e4700 1 -- 172.21.15.49:0/531711237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14c0045ad0 con 0x7f14c00f5410 2023-11-30T21:47:30.414 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.414+0000 7f14bcff9700 1 -- 172.21.15.49:0/531711237 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f14b001db70 con 0x7f14c00f5410 2023-11-30T21:47:30.414 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:30.414 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:30.416 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.415+0000 7f14c68e4700 1 -- 172.21.15.49:0/531711237 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f14a802ff60 msgr2=0x7f14a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:30.416 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.415+0000 7f14c68e4700 1 --2- 172.21.15.49:0/531711237 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f14a802ff60 0x7f14a8032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f14b40096a0 tx=0x7f14b4006b40).stop 2023-11-30T21:47:30.416 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.415+0000 7f14c68e4700 1 -- 172.21.15.49:0/531711237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c00f5410 msgr2=0x7f14c0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:30.416 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.415+0000 7f14c68e4700 1 --2- 172.21.15.49:0/531711237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c00f5410 0x7f14c0109450 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f14b0013510 tx=0x7f14b000b040).stop 2023-11-30T21:47:30.416 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.415+0000 7f14c68e4700 1 -- 172.21.15.49:0/531711237 shutdown_connections 2023-11-30T21:47:30.416 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.415+0000 7f14c68e4700 1 --2- 172.21.15.49:0/531711237 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f14a802ff60 0x7f14a8032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:30.416 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.415+0000 7f14c68e4700 1 --2- 172.21.15.49:0/531711237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14c00f5410 0x7f14c0109450 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:30.416 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.415+0000 7f14c68e4700 1 -- 172.21.15.49:0/531711237 >> 172.21.15.49:0/531711237 conn(0x7f14c00f0b70 msgr2=0x7f14c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:30.417 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.415+0000 7f14c68e4700 1 -- 172.21.15.49:0/531711237 shutdown_connections 2023-11-30T21:47:30.417 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:30.415+0000 7f14c68e4700 1 -- 172.21.15.49:0/531711237 wait complete. 2023-11-30T21:47:30.417 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:31.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:29.489467+0000 mgr.smithi027.brrjcl (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:31.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:30.414009+0000 mon.smithi027 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.49:0/531711237' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:31.806 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:31.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:32.986 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.985+0000 7fad5e581700 1 -- 172.21.15.49:0/928309530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad580f5410 msgr2=0x7fad580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:32.986 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.985+0000 7fad5e581700 1 --2- 172.21.15.49:0/928309530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad580f5410 0x7fad580f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fad48009fb0 tx=0x7fad48009b30).stop 2023-11-30T21:47:32.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.985+0000 7fad5e581700 1 -- 172.21.15.49:0/928309530 shutdown_connections 2023-11-30T21:47:32.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.985+0000 7fad5e581700 1 --2- 172.21.15.49:0/928309530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad580f5410 0x7fad580f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:32.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.985+0000 7fad5e581700 1 -- 172.21.15.49:0/928309530 >> 172.21.15.49:0/928309530 conn(0x7fad580f0b70 msgr2=0x7fad580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:32.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.985+0000 7fad5e581700 1 -- 172.21.15.49:0/928309530 shutdown_connections 2023-11-30T21:47:32.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.985+0000 7fad5e581700 1 -- 172.21.15.49:0/928309530 wait complete. 2023-11-30T21:47:32.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.985+0000 7fad5e581700 1 Processor -- start 2023-11-30T21:47:32.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.985+0000 7fad5e581700 1 -- start start 2023-11-30T21:47:32.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.986+0000 7fad5e581700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad580f5410 0x7fad581066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:32.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.986+0000 7fad5e581700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad58106bc0 con 0x7fad580f5410 2023-11-30T21:47:32.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.986+0000 7fad57fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad580f5410 0x7fad581066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:32.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.986+0000 7fad57fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad580f5410 0x7fad581066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:50878/0 (socket says 172.21.15.49:50878) 2023-11-30T21:47:32.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.986+0000 7fad57fff700 1 -- 172.21.15.49:0/3724473780 learned_addr learned my addr 172.21.15.49:0/3724473780 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:32.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.986+0000 7fad57fff700 1 -- 172.21.15.49:0/3724473780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad48005040 con 0x7fad580f5410 2023-11-30T21:47:32.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.986+0000 7fad57fff700 1 --2- 172.21.15.49:0/3724473780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad580f5410 0x7fad581066d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fad48013510 tx=0x7fad4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:32.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.986+0000 7fad54ff9700 1 -- 172.21.15.49:0/3724473780 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad48009320 con 0x7fad580f5410 2023-11-30T21:47:32.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.986+0000 7fad5e581700 1 -- 172.21.15.49:0/3724473780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad58106dc0 con 0x7fad580f5410 2023-11-30T21:47:32.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.986+0000 7fad54ff9700 1 -- 172.21.15.49:0/3724473780 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad48013f00 con 0x7fad580f5410 2023-11-30T21:47:32.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.986+0000 7fad54ff9700 1 -- 172.21.15.49:0/3724473780 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad48014d90 con 0x7fad580f5410 2023-11-30T21:47:32.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.986+0000 7fad5e581700 1 -- 172.21.15.49:0/3724473780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad58107250 con 0x7fad580f5410 2023-11-30T21:47:32.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.987+0000 7fad5e581700 1 -- 172.21.15.49:0/3724473780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad58045ad0 con 0x7fad580f5410 2023-11-30T21:47:32.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.987+0000 7fad54ff9700 1 -- 172.21.15.49:0/3724473780 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad480095f0 con 0x7fad580f5410 2023-11-30T21:47:32.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.987+0000 7fad54ff9700 1 --2- 172.21.15.49:0/3724473780 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fad4002ff60 0x7fad40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:32.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.987+0000 7fad54ff9700 1 -- 172.21.15.49:0/3724473780 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad48041080 con 0x7fad580f5410 2023-11-30T21:47:32.989 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.988+0000 7fad577fe700 1 --2- 172.21.15.49:0/3724473780 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fad4002ff60 0x7fad40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:32.990 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.988+0000 7fad577fe700 1 --2- 172.21.15.49:0/3724473780 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fad4002ff60 0x7fad40032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fad4c0096a0 tx=0x7fad4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:32.992+0000 7fad54ff9700 1 -- 172.21.15.49:0/3724473780 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad48014420 con 0x7fad580f5410 2023-11-30T21:47:33.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:33.213+0000 7fad5e581700 1 -- 172.21.15.49:0/3724473780 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad580fae60 con 0x7fad580f5410 2023-11-30T21:47:33.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:33.214+0000 7fad54ff9700 1 -- 172.21.15.49:0/3724473780 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad48010230 con 0x7fad580f5410 2023-11-30T21:47:33.215 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:33.215 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:33.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:33.216+0000 7fad5e581700 1 -- 172.21.15.49:0/3724473780 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fad4002ff60 msgr2=0x7fad40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:33.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:33.216+0000 7fad5e581700 1 --2- 172.21.15.49:0/3724473780 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fad4002ff60 0x7fad40032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fad4c0096a0 tx=0x7fad4c006b40).stop 2023-11-30T21:47:33.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:33.216+0000 7fad5e581700 1 -- 172.21.15.49:0/3724473780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad580f5410 msgr2=0x7fad581066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:33.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:33.216+0000 7fad5e581700 1 --2- 172.21.15.49:0/3724473780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad580f5410 0x7fad581066d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fad48013510 tx=0x7fad4800b040).stop 2023-11-30T21:47:33.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:33.216+0000 7fad5e581700 1 -- 172.21.15.49:0/3724473780 shutdown_connections 2023-11-30T21:47:33.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:33.216+0000 7fad5e581700 1 --2- 172.21.15.49:0/3724473780 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fad4002ff60 0x7fad40032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:33.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:33.216+0000 7fad5e581700 1 --2- 172.21.15.49:0/3724473780 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad580f5410 0x7fad581066d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:33.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:33.216+0000 7fad5e581700 1 -- 172.21.15.49:0/3724473780 >> 172.21.15.49:0/3724473780 conn(0x7fad580f0b70 msgr2=0x7fad5805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:33.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:33.216+0000 7fad5e581700 1 -- 172.21.15.49:0/3724473780 shutdown_connections 2023-11-30T21:47:33.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:33.216+0000 7fad5e581700 1 -- 172.21.15.49:0/3724473780 wait complete. 2023-11-30T21:47:33.218 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:33.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:33 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:31.489803+0000 mgr.smithi027.brrjcl (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:34.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:34 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:33.214773+0000 mon.smithi027 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.49:0/3724473780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:34.612 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:34.613 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:35.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:35 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:33.490171+0000 mgr.smithi027.brrjcl (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:35.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.797+0000 7f6e970bf700 1 -- 172.21.15.49:0/2603271747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e900f6510 msgr2=0x7f6e900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:35.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.797+0000 7f6e970bf700 1 --2- 172.21.15.49:0/2603271747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e900f6510 0x7f6e900f6930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f6e84009fb0 tx=0x7f6e84009b30).stop 2023-11-30T21:47:35.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.798+0000 7f6e970bf700 1 -- 172.21.15.49:0/2603271747 shutdown_connections 2023-11-30T21:47:35.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.798+0000 7f6e970bf700 1 --2- 172.21.15.49:0/2603271747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e900f6510 0x7f6e900f6930 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:35.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.798+0000 7f6e970bf700 1 -- 172.21.15.49:0/2603271747 >> 172.21.15.49:0/2603271747 conn(0x7f6e900f1c70 msgr2=0x7f6e900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:35.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.798+0000 7f6e970bf700 1 -- 172.21.15.49:0/2603271747 shutdown_connections 2023-11-30T21:47:35.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.798+0000 7f6e970bf700 1 -- 172.21.15.49:0/2603271747 wait complete. 2023-11-30T21:47:35.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.798+0000 7f6e970bf700 1 Processor -- start 2023-11-30T21:47:35.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.798+0000 7f6e970bf700 1 -- start start 2023-11-30T21:47:35.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.799+0000 7f6e970bf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e900f6510 0x7f6e90106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:35.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.799+0000 7f6e970bf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e90106c20 con 0x7f6e900f6510 2023-11-30T21:47:35.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.799+0000 7f6e94e5b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e900f6510 0x7f6e90106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:35.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.799+0000 7f6e94e5b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e900f6510 0x7f6e90106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:50900/0 (socket says 172.21.15.49:50900) 2023-11-30T21:47:35.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.799+0000 7f6e94e5b700 1 -- 172.21.15.49:0/280763597 learned_addr learned my addr 172.21.15.49:0/280763597 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:35.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.799+0000 7f6e94e5b700 1 -- 172.21.15.49:0/280763597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e84005040 con 0x7f6e900f6510 2023-11-30T21:47:35.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.799+0000 7f6e94e5b700 1 --2- 172.21.15.49:0/280763597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e900f6510 0x7f6e90106730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f6e84013510 tx=0x7f6e8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:35.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.799+0000 7f6e8d7fa700 1 -- 172.21.15.49:0/280763597 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e84009320 con 0x7f6e900f6510 2023-11-30T21:47:35.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.800+0000 7f6e970bf700 1 -- 172.21.15.49:0/280763597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e90106e20 con 0x7f6e900f6510 2023-11-30T21:47:35.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.800+0000 7f6e8d7fa700 1 -- 172.21.15.49:0/280763597 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e84013f00 con 0x7f6e900f6510 2023-11-30T21:47:35.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.800+0000 7f6e8d7fa700 1 -- 172.21.15.49:0/280763597 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e84014d90 con 0x7f6e900f6510 2023-11-30T21:47:35.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.800+0000 7f6e970bf700 1 -- 172.21.15.49:0/280763597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e901072b0 con 0x7f6e900f6510 2023-11-30T21:47:35.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.801+0000 7f6e8d7fa700 1 -- 172.21.15.49:0/280763597 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e840095f0 con 0x7f6e900f6510 2023-11-30T21:47:35.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.801+0000 7f6e970bf700 1 -- 172.21.15.49:0/280763597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e90045ad0 con 0x7f6e900f6510 2023-11-30T21:47:35.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.801+0000 7f6e8d7fa700 1 --2- 172.21.15.49:0/280763597 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6e8002ff60 0x7f6e80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:35.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.801+0000 7f6e8d7fa700 1 -- 172.21.15.49:0/280763597 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e84041000 con 0x7f6e900f6510 2023-11-30T21:47:35.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.801+0000 7f6e8ffff700 1 --2- 172.21.15.49:0/280763597 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6e8002ff60 0x7f6e80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:35.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.801+0000 7f6e8ffff700 1 --2- 172.21.15.49:0/280763597 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6e8002ff60 0x7f6e80032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f6e7c0096a0 tx=0x7f6e7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:35.806 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:35.806+0000 7f6e8d7fa700 1 -- 172.21.15.49:0/280763597 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e84014420 con 0x7f6e900f6510 2023-11-30T21:47:36.023 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:36.023+0000 7f6e970bf700 1 -- 172.21.15.49:0/280763597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e900fbf60 con 0x7f6e900f6510 2023-11-30T21:47:36.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:36.023+0000 7f6e8d7fa700 1 -- 172.21.15.49:0/280763597 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e8401d7f0 con 0x7f6e900f6510 2023-11-30T21:47:36.024 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:36.024 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:36.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:36.025+0000 7f6e970bf700 1 -- 172.21.15.49:0/280763597 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6e8002ff60 msgr2=0x7f6e80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:36.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:36.025+0000 7f6e970bf700 1 --2- 172.21.15.49:0/280763597 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6e8002ff60 0x7f6e80032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f6e7c0096a0 tx=0x7f6e7c006b40).stop 2023-11-30T21:47:36.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:36.025+0000 7f6e970bf700 1 -- 172.21.15.49:0/280763597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e900f6510 msgr2=0x7f6e90106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:36.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:36.025+0000 7f6e970bf700 1 --2- 172.21.15.49:0/280763597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e900f6510 0x7f6e90106730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f6e84013510 tx=0x7f6e8400b040).stop 2023-11-30T21:47:36.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:36.025+0000 7f6e970bf700 1 -- 172.21.15.49:0/280763597 shutdown_connections 2023-11-30T21:47:36.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:36.025+0000 7f6e970bf700 1 --2- 172.21.15.49:0/280763597 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6e8002ff60 0x7f6e80032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:36.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:36.025+0000 7f6e970bf700 1 --2- 172.21.15.49:0/280763597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e900f6510 0x7f6e90106730 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:36.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:36.025+0000 7f6e970bf700 1 -- 172.21.15.49:0/280763597 >> 172.21.15.49:0/280763597 conn(0x7f6e900f1c70 msgr2=0x7f6e900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:36.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:36.025+0000 7f6e970bf700 1 -- 172.21.15.49:0/280763597 shutdown_connections 2023-11-30T21:47:36.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:36.025+0000 7f6e970bf700 1 -- 172.21.15.49:0/280763597 wait complete. 2023-11-30T21:47:36.027 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:36.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:36.023820+0000 mon.smithi027 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.49:0/280763597' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:37.414 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:37.415 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:37.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:37 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:35.490610+0000 mgr.smithi027.brrjcl (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:38.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.626+0000 7f63907ad700 1 -- 172.21.15.49:0/3617734749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388060220 msgr2=0x7f63880f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:38.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.626+0000 7f63907ad700 1 --2- 172.21.15.49:0/3617734749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388060220 0x7f63880f9150 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f6384009fb0 tx=0x7f6384009b30).stop 2023-11-30T21:47:38.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.626+0000 7f63907ad700 1 -- 172.21.15.49:0/3617734749 shutdown_connections 2023-11-30T21:47:38.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.626+0000 7f63907ad700 1 --2- 172.21.15.49:0/3617734749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388060220 0x7f63880f9150 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:38.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.626+0000 7f63907ad700 1 -- 172.21.15.49:0/3617734749 >> 172.21.15.49:0/3617734749 conn(0x7f63880f0b70 msgr2=0x7f63880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:38.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.626+0000 7f63907ad700 1 -- 172.21.15.49:0/3617734749 shutdown_connections 2023-11-30T21:47:38.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.626+0000 7f63907ad700 1 -- 172.21.15.49:0/3617734749 wait complete. 2023-11-30T21:47:38.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.627+0000 7f63907ad700 1 Processor -- start 2023-11-30T21:47:38.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.627+0000 7f63907ad700 1 -- start start 2023-11-30T21:47:38.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.627+0000 7f63907ad700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388060220 0x7f63880f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:38.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.627+0000 7f63907ad700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63880f5050 con 0x7f6388060220 2023-11-30T21:47:38.629 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.627+0000 7f638e549700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388060220 0x7f63880f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:38.629 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.627+0000 7f638e549700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388060220 0x7f63880f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:37824/0 (socket says 172.21.15.49:37824) 2023-11-30T21:47:38.629 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.627+0000 7f638e549700 1 -- 172.21.15.49:0/2888527440 learned_addr learned my addr 172.21.15.49:0/2888527440 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:38.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.627+0000 7f638e549700 1 -- 172.21.15.49:0/2888527440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6384005040 con 0x7f6388060220 2023-11-30T21:47:38.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.628+0000 7f638e549700 1 --2- 172.21.15.49:0/2888527440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388060220 0x7f63880f6bd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f6384013510 tx=0x7f638400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:38.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.628+0000 7f637effd700 1 -- 172.21.15.49:0/2888527440 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6384009320 con 0x7f6388060220 2023-11-30T21:47:38.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.628+0000 7f63907ad700 1 -- 172.21.15.49:0/2888527440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63880f51f0 con 0x7f6388060220 2023-11-30T21:47:38.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.628+0000 7f637effd700 1 -- 172.21.15.49:0/2888527440 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6384013f00 con 0x7f6388060220 2023-11-30T21:47:38.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.628+0000 7f637effd700 1 -- 172.21.15.49:0/2888527440 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6384014dc0 con 0x7f6388060220 2023-11-30T21:47:38.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.628+0000 7f63907ad700 1 -- 172.21.15.49:0/2888527440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63880f5680 con 0x7f6388060220 2023-11-30T21:47:38.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.629+0000 7f637effd700 1 -- 172.21.15.49:0/2888527440 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f638401d3d0 con 0x7f6388060220 2023-11-30T21:47:38.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.629+0000 7f637effd700 1 --2- 172.21.15.49:0/2888527440 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f637402ff60 0x7f6374032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:38.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.629+0000 7f637effd700 1 -- 172.21.15.49:0/2888527440 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6384041110 con 0x7f6388060220 2023-11-30T21:47:38.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.629+0000 7f63907ad700 1 -- 172.21.15.49:0/2888527440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6388059100 con 0x7f6388060220 2023-11-30T21:47:38.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.629+0000 7f638dd48700 1 --2- 172.21.15.49:0/2888527440 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f637402ff60 0x7f6374032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:38.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.630+0000 7f638dd48700 1 --2- 172.21.15.49:0/2888527440 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f637402ff60 0x7f6374032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f63780096a0 tx=0x7f6378006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:38.634 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.634+0000 7f637effd700 1 -- 172.21.15.49:0/2888527440 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6384012920 con 0x7f6388060220 2023-11-30T21:47:38.857 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.856+0000 7f63907ad700 1 -- 172.21.15.49:0/2888527440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63880f5d80 con 0x7f6388060220 2023-11-30T21:47:38.857 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.857+0000 7f637effd700 1 -- 172.21.15.49:0/2888527440 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f638402b020 con 0x7f6388060220 2023-11-30T21:47:38.858 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:38.858 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:38.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.858+0000 7f63907ad700 1 -- 172.21.15.49:0/2888527440 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f637402ff60 msgr2=0x7f6374032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:38.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.858+0000 7f63907ad700 1 --2- 172.21.15.49:0/2888527440 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f637402ff60 0x7f6374032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f63780096a0 tx=0x7f6378006b40).stop 2023-11-30T21:47:38.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.858+0000 7f63907ad700 1 -- 172.21.15.49:0/2888527440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388060220 msgr2=0x7f63880f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:38.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.858+0000 7f63907ad700 1 --2- 172.21.15.49:0/2888527440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388060220 0x7f63880f6bd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f6384013510 tx=0x7f638400b040).stop 2023-11-30T21:47:38.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.858+0000 7f63907ad700 1 -- 172.21.15.49:0/2888527440 shutdown_connections 2023-11-30T21:47:38.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.858+0000 7f63907ad700 1 --2- 172.21.15.49:0/2888527440 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f637402ff60 0x7f6374032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:38.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.858+0000 7f63907ad700 1 --2- 172.21.15.49:0/2888527440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6388060220 0x7f63880f6bd0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:38.860 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.858+0000 7f63907ad700 1 -- 172.21.15.49:0/2888527440 >> 172.21.15.49:0/2888527440 conn(0x7f63880f0b70 msgr2=0x7f63880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:38.860 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.859+0000 7f63907ad700 1 -- 172.21.15.49:0/2888527440 shutdown_connections 2023-11-30T21:47:38.860 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:38.859+0000 7f63907ad700 1 -- 172.21.15.49:0/2888527440 wait complete. 2023-11-30T21:47:38.860 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:39.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:37.491075+0000 mgr.smithi027.brrjcl (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:39.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:38.857511+0000 mon.smithi027 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.49:0/2888527440' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:40.274 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:40.275 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:41.500 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.500+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3141178777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b00f62a0 msgr2=0x7fa6b00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:41.500 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.500+0000 7fa6b65f8700 1 --2- 172.21.15.49:0/3141178777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b00f62a0 0x7fa6b00f66c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fa6ac009fb0 tx=0x7fa6ac009b30).stop 2023-11-30T21:47:41.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.500+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3141178777 shutdown_connections 2023-11-30T21:47:41.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.500+0000 7fa6b65f8700 1 --2- 172.21.15.49:0/3141178777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b00f62a0 0x7fa6b00f66c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:41.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.500+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3141178777 >> 172.21.15.49:0/3141178777 conn(0x7fa6b00f1aa0 msgr2=0x7fa6b00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:41.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.500+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3141178777 shutdown_connections 2023-11-30T21:47:41.501 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.500+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3141178777 wait complete. 2023-11-30T21:47:41.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.500+0000 7fa6b65f8700 1 Processor -- start 2023-11-30T21:47:41.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.500+0000 7fa6b65f8700 1 -- start start 2023-11-30T21:47:41.502 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.500+0000 7fa6b65f8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b00f62a0 0x7fa6b01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:41.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.500+0000 7fa6b65f8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b01069a0 con 0x7fa6b00f62a0 2023-11-30T21:47:41.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.501+0000 7fa6b55f6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b00f62a0 0x7fa6b01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:41.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.501+0000 7fa6b55f6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b00f62a0 0x7fa6b01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:37832/0 (socket says 172.21.15.49:37832) 2023-11-30T21:47:41.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.501+0000 7fa6b55f6700 1 -- 172.21.15.49:0/3292533947 learned_addr learned my addr 172.21.15.49:0/3292533947 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:41.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.501+0000 7fa6b55f6700 1 -- 172.21.15.49:0/3292533947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac005040 con 0x7fa6b00f62a0 2023-11-30T21:47:41.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.501+0000 7fa6b55f6700 1 --2- 172.21.15.49:0/3292533947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b00f62a0 0x7fa6b01064b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fa6ac013510 tx=0x7fa6ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:41.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.501+0000 7fa6a5ffb700 1 -- 172.21.15.49:0/3292533947 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6ac009320 con 0x7fa6b00f62a0 2023-11-30T21:47:41.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.501+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3292533947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b0106ba0 con 0x7fa6b00f62a0 2023-11-30T21:47:41.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.501+0000 7fa6a5ffb700 1 -- 172.21.15.49:0/3292533947 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6ac013f00 con 0x7fa6b00f62a0 2023-11-30T21:47:41.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.501+0000 7fa6a5ffb700 1 -- 172.21.15.49:0/3292533947 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6ac014dc0 con 0x7fa6b00f62a0 2023-11-30T21:47:41.503 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.502+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3292533947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b0107030 con 0x7fa6b00f62a0 2023-11-30T21:47:41.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.502+0000 7fa6a5ffb700 1 -- 172.21.15.49:0/3292533947 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6ac014420 con 0x7fa6b00f62a0 2023-11-30T21:47:41.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.502+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3292533947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b0045ad0 con 0x7fa6b00f62a0 2023-11-30T21:47:41.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.503+0000 7fa6a5ffb700 1 --2- 172.21.15.49:0/3292533947 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa69c02ff60 0x7fa69c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:41.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.503+0000 7fa6a5ffb700 1 -- 172.21.15.49:0/3292533947 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6ac041030 con 0x7fa6b00f62a0 2023-11-30T21:47:41.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.503+0000 7fa6b4df5700 1 --2- 172.21.15.49:0/3292533947 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa69c02ff60 0x7fa69c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:41.505 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.503+0000 7fa6b4df5700 1 --2- 172.21.15.49:0/3292533947 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa69c02ff60 0x7fa69c032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa6a00096a0 tx=0x7fa6a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:41.508 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.507+0000 7fa6a5ffb700 1 -- 172.21.15.49:0/3292533947 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6ac0095f0 con 0x7fa6b00f62a0 2023-11-30T21:47:41.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:41 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:39.491408+0000 mgr.smithi027.brrjcl (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:41.725 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.725+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3292533947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6b0107ee0 con 0x7fa6b00f62a0 2023-11-30T21:47:41.726 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.726+0000 7fa6a5ffb700 1 -- 172.21.15.49:0/3292533947 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6ac01cb10 con 0x7fa6b00f62a0 2023-11-30T21:47:41.726 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:41.726 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:41.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.727+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3292533947 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa69c02ff60 msgr2=0x7fa69c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:41.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.727+0000 7fa6b65f8700 1 --2- 172.21.15.49:0/3292533947 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa69c02ff60 0x7fa69c032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa6a00096a0 tx=0x7fa6a0006b40).stop 2023-11-30T21:47:41.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.727+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3292533947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b00f62a0 msgr2=0x7fa6b01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:41.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.727+0000 7fa6b65f8700 1 --2- 172.21.15.49:0/3292533947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b00f62a0 0x7fa6b01064b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fa6ac013510 tx=0x7fa6ac00b040).stop 2023-11-30T21:47:41.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.727+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3292533947 shutdown_connections 2023-11-30T21:47:41.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.727+0000 7fa6b65f8700 1 --2- 172.21.15.49:0/3292533947 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa69c02ff60 0x7fa69c032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:41.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.727+0000 7fa6b65f8700 1 --2- 172.21.15.49:0/3292533947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b00f62a0 0x7fa6b01064b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:41.728 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.727+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3292533947 >> 172.21.15.49:0/3292533947 conn(0x7fa6b00f1aa0 msgr2=0x7fa6b00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:41.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.727+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3292533947 shutdown_connections 2023-11-30T21:47:41.729 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:41.727+0000 7fa6b65f8700 1 -- 172.21.15.49:0/3292533947 wait complete. 2023-11-30T21:47:41.729 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:42.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:42 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:41.726083+0000 mon.smithi027 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.49:0/3292533947' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:43.162 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:43.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:43.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:41.491820+0000 mgr.smithi027.brrjcl (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:44.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.361+0000 7f17cea12700 1 -- 172.21.15.49:0/1894783611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c80f6510 msgr2=0x7f17c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:44.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.361+0000 7f17cea12700 1 --2- 172.21.15.49:0/1894783611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c80f6510 0x7f17c80f6930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f17b8009fb0 tx=0x7f17b8009b30).stop 2023-11-30T21:47:44.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.361+0000 7f17cea12700 1 -- 172.21.15.49:0/1894783611 shutdown_connections 2023-11-30T21:47:44.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.361+0000 7f17cea12700 1 --2- 172.21.15.49:0/1894783611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c80f6510 0x7f17c80f6930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:44.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.361+0000 7f17cea12700 1 -- 172.21.15.49:0/1894783611 >> 172.21.15.49:0/1894783611 conn(0x7f17c80f1c70 msgr2=0x7f17c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:44.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.361+0000 7f17cea12700 1 -- 172.21.15.49:0/1894783611 shutdown_connections 2023-11-30T21:47:44.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.361+0000 7f17cea12700 1 -- 172.21.15.49:0/1894783611 wait complete. 2023-11-30T21:47:44.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.362+0000 7f17cea12700 1 Processor -- start 2023-11-30T21:47:44.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.362+0000 7f17cea12700 1 -- start start 2023-11-30T21:47:44.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.362+0000 7f17cea12700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c80f6510 0x7f17c8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:44.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.362+0000 7f17cea12700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c8106c20 con 0x7f17c80f6510 2023-11-30T21:47:44.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.362+0000 7f17c7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c80f6510 0x7f17c8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.362+0000 7f17c7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c80f6510 0x7f17c8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:37850/0 (socket says 172.21.15.49:37850) 2023-11-30T21:47:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.362+0000 7f17c7fff700 1 -- 172.21.15.49:0/2938429506 learned_addr learned my addr 172.21.15.49:0/2938429506 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.363+0000 7f17c7fff700 1 -- 172.21.15.49:0/2938429506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b8005040 con 0x7f17c80f6510 2023-11-30T21:47:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.363+0000 7f17c7fff700 1 --2- 172.21.15.49:0/2938429506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c80f6510 0x7f17c8106730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f17b8013510 tx=0x7f17b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.363+0000 7f17c4ff9700 1 -- 172.21.15.49:0/2938429506 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17b8009320 con 0x7f17c80f6510 2023-11-30T21:47:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.363+0000 7f17cea12700 1 -- 172.21.15.49:0/2938429506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c8106e20 con 0x7f17c80f6510 2023-11-30T21:47:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.363+0000 7f17c4ff9700 1 -- 172.21.15.49:0/2938429506 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17b8013f00 con 0x7f17c80f6510 2023-11-30T21:47:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.363+0000 7f17c4ff9700 1 -- 172.21.15.49:0/2938429506 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17b8014d90 con 0x7f17c80f6510 2023-11-30T21:47:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.363+0000 7f17cea12700 1 -- 172.21.15.49:0/2938429506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c81072b0 con 0x7f17c80f6510 2023-11-30T21:47:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.364+0000 7f17c4ff9700 1 -- 172.21.15.49:0/2938429506 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f17b80095f0 con 0x7f17c80f6510 2023-11-30T21:47:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.364+0000 7f17cea12700 1 -- 172.21.15.49:0/2938429506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c8045ad0 con 0x7f17c80f6510 2023-11-30T21:47:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.364+0000 7f17c4ff9700 1 --2- 172.21.15.49:0/2938429506 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f17b002ff60 0x7f17b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.364+0000 7f17c4ff9700 1 -- 172.21.15.49:0/2938429506 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17b80410a0 con 0x7f17c80f6510 2023-11-30T21:47:44.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.364+0000 7f17c77fe700 1 --2- 172.21.15.49:0/2938429506 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f17b002ff60 0x7f17b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:44.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.364+0000 7f17c77fe700 1 --2- 172.21.15.49:0/2938429506 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f17b002ff60 0x7f17b0032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f17bc0096a0 tx=0x7f17bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:44.373 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.369+0000 7f17c4ff9700 1 -- 172.21.15.49:0/2938429506 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17b801d7f0 con 0x7f17c80f6510 2023-11-30T21:47:44.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.586+0000 7f17cea12700 1 -- 172.21.15.49:0/2938429506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17c80fbf60 con 0x7f17c80f6510 2023-11-30T21:47:44.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.586+0000 7f17c4ff9700 1 -- 172.21.15.49:0/2938429506 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f17b8010230 con 0x7f17c80f6510 2023-11-30T21:47:44.588 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:44.588 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:44.589 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.588+0000 7f17cea12700 1 -- 172.21.15.49:0/2938429506 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f17b002ff60 msgr2=0x7f17b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:44.589 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.588+0000 7f17cea12700 1 --2- 172.21.15.49:0/2938429506 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f17b002ff60 0x7f17b0032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f17bc0096a0 tx=0x7f17bc006b40).stop 2023-11-30T21:47:44.589 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.588+0000 7f17cea12700 1 -- 172.21.15.49:0/2938429506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c80f6510 msgr2=0x7f17c8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:44.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.588+0000 7f17cea12700 1 --2- 172.21.15.49:0/2938429506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c80f6510 0x7f17c8106730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f17b8013510 tx=0x7f17b800b040).stop 2023-11-30T21:47:44.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.588+0000 7f17cea12700 1 -- 172.21.15.49:0/2938429506 shutdown_connections 2023-11-30T21:47:44.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.588+0000 7f17cea12700 1 --2- 172.21.15.49:0/2938429506 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f17b002ff60 0x7f17b0032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:44.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.588+0000 7f17cea12700 1 --2- 172.21.15.49:0/2938429506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17c80f6510 0x7f17c8106730 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:44.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.588+0000 7f17cea12700 1 -- 172.21.15.49:0/2938429506 >> 172.21.15.49:0/2938429506 conn(0x7f17c80f1c70 msgr2=0x7f17c80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:44.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.588+0000 7f17cea12700 1 -- 172.21.15.49:0/2938429506 shutdown_connections 2023-11-30T21:47:44.590 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:44.588+0000 7f17cea12700 1 -- 172.21.15.49:0/2938429506 wait complete. 2023-11-30T21:47:44.590 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:45.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:45 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:43.492174+0000 mgr.smithi027.brrjcl (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:45 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:44.587946+0000 mon.smithi027 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.49:0/2938429506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:46.009 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:46.009 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.202+0000 7fb74ac0d700 1 -- 172.21.15.49:0/3457796139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7440f51a0 msgr2=0x7fb7440f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.202+0000 7fb74ac0d700 1 --2- 172.21.15.49:0/3457796139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7440f51a0 0x7fb7440f55c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fb740009fb0 tx=0x7fb740009b30).stop 2023-11-30T21:47:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.202+0000 7fb74ac0d700 1 -- 172.21.15.49:0/3457796139 shutdown_connections 2023-11-30T21:47:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.202+0000 7fb74ac0d700 1 --2- 172.21.15.49:0/3457796139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7440f51a0 0x7fb7440f55c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.202+0000 7fb74ac0d700 1 -- 172.21.15.49:0/3457796139 >> 172.21.15.49:0/3457796139 conn(0x7fb7440f09a0 msgr2=0x7fb7440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.202+0000 7fb74ac0d700 1 -- 172.21.15.49:0/3457796139 shutdown_connections 2023-11-30T21:47:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.202+0000 7fb74ac0d700 1 -- 172.21.15.49:0/3457796139 wait complete. 2023-11-30T21:47:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.202+0000 7fb74ac0d700 1 Processor -- start 2023-11-30T21:47:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.203+0000 7fb74ac0d700 1 -- start start 2023-11-30T21:47:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.203+0000 7fb74ac0d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7440f51a0 0x7fb7441091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.203+0000 7fb74ac0d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb744060220 con 0x7fb7440f51a0 2023-11-30T21:47:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.203+0000 7fb749c0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7440f51a0 0x7fb7441091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.203+0000 7fb749c0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7440f51a0 0x7fb7441091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:37868/0 (socket says 172.21.15.49:37868) 2023-11-30T21:47:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.203+0000 7fb749c0b700 1 -- 172.21.15.49:0/2539670551 learned_addr learned my addr 172.21.15.49:0/2539670551 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.203+0000 7fb749c0b700 1 -- 172.21.15.49:0/2539670551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb740005040 con 0x7fb7440f51a0 2023-11-30T21:47:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.203+0000 7fb749c0b700 1 --2- 172.21.15.49:0/2539670551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7440f51a0 0x7fb7441091d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fb740013510 tx=0x7fb74000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.203+0000 7fb7367fc700 1 -- 172.21.15.49:0/2539670551 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb740009320 con 0x7fb7440f51a0 2023-11-30T21:47:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.204+0000 7fb74ac0d700 1 -- 172.21.15.49:0/2539670551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb744060420 con 0x7fb7440f51a0 2023-11-30T21:47:47.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.204+0000 7fb7367fc700 1 -- 172.21.15.49:0/2539670551 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb740013f00 con 0x7fb7440f51a0 2023-11-30T21:47:47.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.204+0000 7fb7367fc700 1 -- 172.21.15.49:0/2539670551 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb740014dc0 con 0x7fb7440f51a0 2023-11-30T21:47:47.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.204+0000 7fb74ac0d700 1 -- 172.21.15.49:0/2539670551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb744109880 con 0x7fb7440f51a0 2023-11-30T21:47:47.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.204+0000 7fb7367fc700 1 -- 172.21.15.49:0/2539670551 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb740014420 con 0x7fb7440f51a0 2023-11-30T21:47:47.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.205+0000 7fb74ac0d700 1 -- 172.21.15.49:0/2539670551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb744059100 con 0x7fb7440f51a0 2023-11-30T21:47:47.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.205+0000 7fb7367fc700 1 --2- 172.21.15.49:0/2539670551 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb73002ff60 0x7fb730032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:47.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.205+0000 7fb7367fc700 1 -- 172.21.15.49:0/2539670551 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb740041020 con 0x7fb7440f51a0 2023-11-30T21:47:47.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.205+0000 7fb74940a700 1 --2- 172.21.15.49:0/2539670551 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb73002ff60 0x7fb730032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:47.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.205+0000 7fb74940a700 1 --2- 172.21.15.49:0/2539670551 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb73002ff60 0x7fb730032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fb7380096a0 tx=0x7fb738006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:47.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.209+0000 7fb7367fc700 1 -- 172.21.15.49:0/2539670551 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7400095f0 con 0x7fb7440f51a0 2023-11-30T21:47:47.428 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.426+0000 7fb74ac0d700 1 -- 172.21.15.49:0/2539670551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb7440fabe0 con 0x7fb7440f51a0 2023-11-30T21:47:47.428 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.427+0000 7fb7367fc700 1 -- 172.21.15.49:0/2539670551 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb74001cb10 con 0x7fb7440f51a0 2023-11-30T21:47:47.429 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:47.429 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:47.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.429+0000 7fb74ac0d700 1 -- 172.21.15.49:0/2539670551 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb73002ff60 msgr2=0x7fb730032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:47.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.429+0000 7fb74ac0d700 1 --2- 172.21.15.49:0/2539670551 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb73002ff60 0x7fb730032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fb7380096a0 tx=0x7fb738006b40).stop 2023-11-30T21:47:47.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.429+0000 7fb74ac0d700 1 -- 172.21.15.49:0/2539670551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7440f51a0 msgr2=0x7fb7441091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:47.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.429+0000 7fb74ac0d700 1 --2- 172.21.15.49:0/2539670551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7440f51a0 0x7fb7441091d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fb740013510 tx=0x7fb74000b040).stop 2023-11-30T21:47:47.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.429+0000 7fb74ac0d700 1 -- 172.21.15.49:0/2539670551 shutdown_connections 2023-11-30T21:47:47.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.429+0000 7fb74ac0d700 1 --2- 172.21.15.49:0/2539670551 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb73002ff60 0x7fb730032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:47.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.429+0000 7fb74ac0d700 1 --2- 172.21.15.49:0/2539670551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7440f51a0 0x7fb7441091d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:47.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.429+0000 7fb74ac0d700 1 -- 172.21.15.49:0/2539670551 >> 172.21.15.49:0/2539670551 conn(0x7fb7440f09a0 msgr2=0x7fb7440f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:47.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.429+0000 7fb74ac0d700 1 -- 172.21.15.49:0/2539670551 shutdown_connections 2023-11-30T21:47:47.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:47.429+0000 7fb74ac0d700 1 -- 172.21.15.49:0/2539670551 wait complete. 2023-11-30T21:47:47.431 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:47.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:47 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:45.492456+0000 mgr.smithi027.brrjcl (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:48.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:48 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:47.428723+0000 mon.smithi027 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.49:0/2539670551' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:48.803 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:48.803 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:49.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:49 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:47.492864+0000 mgr.smithi027.brrjcl (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:50.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.030+0000 7f772f431700 1 -- 172.21.15.49:0/270651355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77280f51a0 msgr2=0x7f77280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:50.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.030+0000 7f772f431700 1 --2- 172.21.15.49:0/270651355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77280f51a0 0x7f77280f55c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f7724009fb0 tx=0x7f7724009b30).stop 2023-11-30T21:47:50.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.030+0000 7f772f431700 1 -- 172.21.15.49:0/270651355 shutdown_connections 2023-11-30T21:47:50.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.030+0000 7f772f431700 1 --2- 172.21.15.49:0/270651355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77280f51a0 0x7f77280f55c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:50.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.030+0000 7f772f431700 1 -- 172.21.15.49:0/270651355 >> 172.21.15.49:0/270651355 conn(0x7f77280f09a0 msgr2=0x7f77280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:50.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.030+0000 7f772f431700 1 -- 172.21.15.49:0/270651355 shutdown_connections 2023-11-30T21:47:50.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.030+0000 7f772f431700 1 -- 172.21.15.49:0/270651355 wait complete. 2023-11-30T21:47:50.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.030+0000 7f772f431700 1 Processor -- start 2023-11-30T21:47:50.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.030+0000 7f772f431700 1 -- start start 2023-11-30T21:47:50.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.030+0000 7f772f431700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77280f51a0 0x7f7728109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:50.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.030+0000 7f772f431700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7728060280 con 0x7f77280f51a0 2023-11-30T21:47:50.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.031+0000 7f772e42f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77280f51a0 0x7f7728109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:50.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.031+0000 7f772e42f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77280f51a0 0x7f7728109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:56596/0 (socket says 172.21.15.49:56596) 2023-11-30T21:47:50.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.031+0000 7f772e42f700 1 -- 172.21.15.49:0/3820880670 learned_addr learned my addr 172.21.15.49:0/3820880670 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:50.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.031+0000 7f772e42f700 1 -- 172.21.15.49:0/3820880670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7724005040 con 0x7f77280f51a0 2023-11-30T21:47:50.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.031+0000 7f772e42f700 1 --2- 172.21.15.49:0/3820880670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77280f51a0 0x7f7728109170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f7724013510 tx=0x7f772400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:50.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.031+0000 7f771effd700 1 -- 172.21.15.49:0/3820880670 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7724009320 con 0x7f77280f51a0 2023-11-30T21:47:50.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.031+0000 7f772f431700 1 -- 172.21.15.49:0/3820880670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7728060480 con 0x7f77280f51a0 2023-11-30T21:47:50.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.031+0000 7f771effd700 1 -- 172.21.15.49:0/3820880670 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7724013f00 con 0x7f77280f51a0 2023-11-30T21:47:50.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.031+0000 7f771effd700 1 -- 172.21.15.49:0/3820880670 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7724014dc0 con 0x7f77280f51a0 2023-11-30T21:47:50.035 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.031+0000 7f772f431700 1 -- 172.21.15.49:0/3820880670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77281098a0 con 0x7f77280f51a0 2023-11-30T21:47:50.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.032+0000 7f771effd700 1 -- 172.21.15.49:0/3820880670 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77240095f0 con 0x7f77280f51a0 2023-11-30T21:47:50.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.032+0000 7f772f431700 1 -- 172.21.15.49:0/3820880670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7728045ad0 con 0x7f77280f51a0 2023-11-30T21:47:50.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.032+0000 7f771effd700 1 --2- 172.21.15.49:0/3820880670 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f771402ff60 0x7f7714032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:50.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.032+0000 7f771effd700 1 -- 172.21.15.49:0/3820880670 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7724041020 con 0x7f77280f51a0 2023-11-30T21:47:50.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.032+0000 7f772dc2e700 1 --2- 172.21.15.49:0/3820880670 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f771402ff60 0x7f7714032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:50.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.033+0000 7f772dc2e700 1 --2- 172.21.15.49:0/3820880670 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f771402ff60 0x7f7714032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f77180096a0 tx=0x7f7718006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:50.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.037+0000 7f771effd700 1 -- 172.21.15.49:0/3820880670 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7724014420 con 0x7f77280f51a0 2023-11-30T21:47:50.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.255+0000 7f772f431700 1 -- 172.21.15.49:0/3820880670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77280fabe0 con 0x7f77280f51a0 2023-11-30T21:47:50.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.256+0000 7f771effd700 1 -- 172.21.15.49:0/3820880670 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f772401c920 con 0x7f77280f51a0 2023-11-30T21:47:50.258 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:50.259 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.257+0000 7f772f431700 1 -- 172.21.15.49:0/3820880670 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f771402ff60 msgr2=0x7f7714032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.257+0000 7f772f431700 1 --2- 172.21.15.49:0/3820880670 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f771402ff60 0x7f7714032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f77180096a0 tx=0x7f7718006b40).stop 2023-11-30T21:47:50.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.257+0000 7f772f431700 1 -- 172.21.15.49:0/3820880670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77280f51a0 msgr2=0x7f7728109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.257+0000 7f772f431700 1 --2- 172.21.15.49:0/3820880670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77280f51a0 0x7f7728109170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f7724013510 tx=0x7f772400b040).stop 2023-11-30T21:47:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.257+0000 7f772f431700 1 -- 172.21.15.49:0/3820880670 shutdown_connections 2023-11-30T21:47:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.257+0000 7f772f431700 1 --2- 172.21.15.49:0/3820880670 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f771402ff60 0x7f7714032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.257+0000 7f772f431700 1 --2- 172.21.15.49:0/3820880670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77280f51a0 0x7f7728109170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.257+0000 7f772f431700 1 -- 172.21.15.49:0/3820880670 >> 172.21.15.49:0/3820880670 conn(0x7f77280f09a0 msgr2=0x7f77280f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.257+0000 7f772f431700 1 -- 172.21.15.49:0/3820880670 shutdown_connections 2023-11-30T21:47:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:50.257+0000 7f772f431700 1 -- 172.21.15.49:0/3820880670 wait complete. 2023-11-30T21:47:50.260 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:51.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:49.493291+0000 mgr.smithi027.brrjcl (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:51.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:50.257174+0000 mon.smithi027 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.49:0/3820880670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:51.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:51.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:52.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.870+0000 7f85a5bd9700 1 -- 172.21.15.49:0/251625937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85a0060280 msgr2=0x7f85a00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:52.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.870+0000 7f85a5bd9700 1 --2- 172.21.15.49:0/251625937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85a0060280 0x7f85a00f90f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f8588009fb0 tx=0x7f8588009b30).stop 2023-11-30T21:47:52.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.870+0000 7f85a5bd9700 1 -- 172.21.15.49:0/251625937 shutdown_connections 2023-11-30T21:47:52.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.870+0000 7f85a5bd9700 1 --2- 172.21.15.49:0/251625937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85a0060280 0x7f85a00f90f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:52.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.870+0000 7f85a5bd9700 1 -- 172.21.15.49:0/251625937 >> 172.21.15.49:0/251625937 conn(0x7f85a00f0b70 msgr2=0x7f85a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:52.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.870+0000 7f85a5bd9700 1 -- 172.21.15.49:0/251625937 shutdown_connections 2023-11-30T21:47:52.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.870+0000 7f85a5bd9700 1 -- 172.21.15.49:0/251625937 wait complete. 2023-11-30T21:47:52.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.871+0000 7f85a5bd9700 1 Processor -- start 2023-11-30T21:47:52.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.871+0000 7f85a5bd9700 1 -- start start 2023-11-30T21:47:52.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.871+0000 7f85a5bd9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85a0060280 0x7f85a01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:52.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.871+0000 7f85a5bd9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a0106bd0 con 0x7f85a0060280 2023-11-30T21:47:52.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.872+0000 7f859f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85a0060280 0x7f85a01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:52.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.872+0000 7f859f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85a0060280 0x7f85a01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:56608/0 (socket says 172.21.15.49:56608) 2023-11-30T21:47:52.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.872+0000 7f859f7fe700 1 -- 172.21.15.49:0/3100766277 learned_addr learned my addr 172.21.15.49:0/3100766277 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:52.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.872+0000 7f859f7fe700 1 -- 172.21.15.49:0/3100766277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8588005040 con 0x7f85a0060280 2023-11-30T21:47:52.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.872+0000 7f859f7fe700 1 --2- 172.21.15.49:0/3100766277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85a0060280 0x7f85a01066e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f8588013510 tx=0x7f858800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:52.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.872+0000 7f8597fff700 1 -- 172.21.15.49:0/3100766277 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8588009320 con 0x7f85a0060280 2023-11-30T21:47:52.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.872+0000 7f85a5bd9700 1 -- 172.21.15.49:0/3100766277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a0106dd0 con 0x7f85a0060280 2023-11-30T21:47:52.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.872+0000 7f85a5bd9700 1 -- 172.21.15.49:0/3100766277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a0107260 con 0x7f85a0060280 2023-11-30T21:47:52.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.872+0000 7f8597fff700 1 -- 172.21.15.49:0/3100766277 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8588013f00 con 0x7f85a0060280 2023-11-30T21:47:52.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.872+0000 7f8597fff700 1 -- 172.21.15.49:0/3100766277 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8588014d90 con 0x7f85a0060280 2023-11-30T21:47:52.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.873+0000 7f8597fff700 1 -- 172.21.15.49:0/3100766277 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8588014420 con 0x7f85a0060280 2023-11-30T21:47:52.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.873+0000 7f85a5bd9700 1 -- 172.21.15.49:0/3100766277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a0045ad0 con 0x7f85a0060280 2023-11-30T21:47:52.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.873+0000 7f8597fff700 1 --2- 172.21.15.49:0/3100766277 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f858c02ff60 0x7f858c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:52.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.873+0000 7f8597fff700 1 -- 172.21.15.49:0/3100766277 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85880412d0 con 0x7f85a0060280 2023-11-30T21:47:52.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.874+0000 7f859effd700 1 --2- 172.21.15.49:0/3100766277 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f858c02ff60 0x7f858c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:52.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.874+0000 7f859effd700 1 --2- 172.21.15.49:0/3100766277 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f858c02ff60 0x7f858c032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f85900096a0 tx=0x7f8590006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:52.881 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:52.878+0000 7f8597fff700 1 -- 172.21.15.49:0/3100766277 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f858801c320 con 0x7f85a0060280 2023-11-30T21:47:53.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:53.096+0000 7f85a5bd9700 1 -- 172.21.15.49:0/3100766277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85a0108110 con 0x7f85a0060280 2023-11-30T21:47:53.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:53.097+0000 7f8597fff700 1 -- 172.21.15.49:0/3100766277 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8588012b20 con 0x7f85a0060280 2023-11-30T21:47:53.099 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:53.099 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:53.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:53.098+0000 7f85a5bd9700 1 -- 172.21.15.49:0/3100766277 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f858c02ff60 msgr2=0x7f858c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:53.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:53.098+0000 7f85a5bd9700 1 --2- 172.21.15.49:0/3100766277 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f858c02ff60 0x7f858c032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f85900096a0 tx=0x7f8590006b40).stop 2023-11-30T21:47:53.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:53.098+0000 7f85a5bd9700 1 -- 172.21.15.49:0/3100766277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85a0060280 msgr2=0x7f85a01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:53.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:53.098+0000 7f85a5bd9700 1 --2- 172.21.15.49:0/3100766277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85a0060280 0x7f85a01066e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f8588013510 tx=0x7f858800b040).stop 2023-11-30T21:47:53.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:53.098+0000 7f85a5bd9700 1 -- 172.21.15.49:0/3100766277 shutdown_connections 2023-11-30T21:47:53.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:53.098+0000 7f85a5bd9700 1 --2- 172.21.15.49:0/3100766277 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f858c02ff60 0x7f858c032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:53.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:53.098+0000 7f85a5bd9700 1 --2- 172.21.15.49:0/3100766277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85a0060280 0x7f85a01066e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:53.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:53.098+0000 7f85a5bd9700 1 -- 172.21.15.49:0/3100766277 >> 172.21.15.49:0/3100766277 conn(0x7f85a00f0b70 msgr2=0x7f85a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:53.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:53.098+0000 7f85a5bd9700 1 -- 172.21.15.49:0/3100766277 shutdown_connections 2023-11-30T21:47:53.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:53.099+0000 7f85a5bd9700 1 -- 172.21.15.49:0/3100766277 wait complete. 2023-11-30T21:47:53.101 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:53.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:53 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:51.493731+0000 mgr.smithi027.brrjcl (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:53.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:53 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:53.098427+0000 mon.smithi027 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.49:0/3100766277' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:54.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:54.486 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:55.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:53.494063+0000 mgr.smithi027.brrjcl (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:55.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.764+0000 7f1ff3478700 1 -- 172.21.15.49:0/3723948734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec0f47c0 msgr2=0x7f1fec0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:55.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.764+0000 7f1ff3478700 1 --2- 172.21.15.49:0/3723948734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec0f47c0 0x7f1fec0f4be0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f1fe8009fb0 tx=0x7f1fe8009b30).stop 2023-11-30T21:47:55.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.764+0000 7f1ff3478700 1 -- 172.21.15.49:0/3723948734 shutdown_connections 2023-11-30T21:47:55.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.764+0000 7f1ff3478700 1 --2- 172.21.15.49:0/3723948734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec0f47c0 0x7f1fec0f4be0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:55.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.764+0000 7f1ff3478700 1 -- 172.21.15.49:0/3723948734 >> 172.21.15.49:0/3723948734 conn(0x7f1fec0f02e0 msgr2=0x7f1fec0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:55.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.764+0000 7f1ff3478700 1 -- 172.21.15.49:0/3723948734 shutdown_connections 2023-11-30T21:47:55.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.764+0000 7f1ff3478700 1 -- 172.21.15.49:0/3723948734 wait complete. 2023-11-30T21:47:55.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.764+0000 7f1ff3478700 1 Processor -- start 2023-11-30T21:47:55.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.764+0000 7f1ff3478700 1 -- start start 2023-11-30T21:47:55.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.765+0000 7f1ff3478700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec0f47c0 0x7f1fec106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:55.767 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.765+0000 7f1ff3478700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec106bf0 con 0x7f1fec0f47c0 2023-11-30T21:47:55.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.765+0000 7f1ff1214700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec0f47c0 0x7f1fec106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:55.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.765+0000 7f1ff1214700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec0f47c0 0x7f1fec106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:56622/0 (socket says 172.21.15.49:56622) 2023-11-30T21:47:55.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.765+0000 7f1ff1214700 1 -- 172.21.15.49:0/3704495461 learned_addr learned my addr 172.21.15.49:0/3704495461 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:55.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.765+0000 7f1ff1214700 1 -- 172.21.15.49:0/3704495461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe8005040 con 0x7f1fec0f47c0 2023-11-30T21:47:55.768 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.765+0000 7f1ff1214700 1 --2- 172.21.15.49:0/3704495461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec0f47c0 0x7f1fec106700 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f1fe8013510 tx=0x7f1fe800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:55.769 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.765+0000 7f1fe1ffb700 1 -- 172.21.15.49:0/3704495461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fe8009320 con 0x7f1fec0f47c0 2023-11-30T21:47:55.769 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.766+0000 7f1ff3478700 1 -- 172.21.15.49:0/3704495461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fec106df0 con 0x7f1fec0f47c0 2023-11-30T21:47:55.769 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.766+0000 7f1fe1ffb700 1 -- 172.21.15.49:0/3704495461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fe8013f00 con 0x7f1fec0f47c0 2023-11-30T21:47:55.769 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.766+0000 7f1fe1ffb700 1 -- 172.21.15.49:0/3704495461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fe8014d90 con 0x7f1fec0f47c0 2023-11-30T21:47:55.769 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.766+0000 7f1ff3478700 1 -- 172.21.15.49:0/3704495461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fec107280 con 0x7f1fec0f47c0 2023-11-30T21:47:55.770 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.766+0000 7f1fe1ffb700 1 -- 172.21.15.49:0/3704495461 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1fe80263a0 con 0x7f1fec0f47c0 2023-11-30T21:47:55.770 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.767+0000 7f1fe1ffb700 1 --2- 172.21.15.49:0/3704495461 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f1fd802ffb0 0x7f1fd8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:55.770 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.767+0000 7f1fe1ffb700 1 -- 172.21.15.49:0/3704495461 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1fe8040f70 con 0x7f1fec0f47c0 2023-11-30T21:47:55.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.767+0000 7f1ff3478700 1 -- 172.21.15.49:0/3704495461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fec045ad0 con 0x7f1fec0f47c0 2023-11-30T21:47:55.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.767+0000 7f1ff0a13700 1 --2- 172.21.15.49:0/3704495461 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f1fd802ffb0 0x7f1fd8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:55.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.767+0000 7f1ff0a13700 1 --2- 172.21.15.49:0/3704495461 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f1fd802ffb0 0x7f1fd8032470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f1fdc0096a0 tx=0x7f1fdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:55.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.772+0000 7f1fe1ffb700 1 -- 172.21.15.49:0/3704495461 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1fe8012920 con 0x7f1fec0f47c0 2023-11-30T21:47:55.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.990+0000 7f1ff3478700 1 -- 172.21.15.49:0/3704495461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fec107fa0 con 0x7f1fec0f47c0 2023-11-30T21:47:55.992 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.991+0000 7f1fe1ffb700 1 -- 172.21.15.49:0/3704495461 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1fe802b020 con 0x7f1fec0f47c0 2023-11-30T21:47:55.993 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:55.993 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:55.994 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.993+0000 7f1ff3478700 1 -- 172.21.15.49:0/3704495461 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f1fd802ffb0 msgr2=0x7f1fd8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:55.994 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.993+0000 7f1ff3478700 1 --2- 172.21.15.49:0/3704495461 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f1fd802ffb0 0x7f1fd8032470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f1fdc0096a0 tx=0x7f1fdc006b40).stop 2023-11-30T21:47:55.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.993+0000 7f1ff3478700 1 -- 172.21.15.49:0/3704495461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec0f47c0 msgr2=0x7f1fec106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:55.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.993+0000 7f1ff3478700 1 --2- 172.21.15.49:0/3704495461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec0f47c0 0x7f1fec106700 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f1fe8013510 tx=0x7f1fe800b040).stop 2023-11-30T21:47:55.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.993+0000 7f1ff3478700 1 -- 172.21.15.49:0/3704495461 shutdown_connections 2023-11-30T21:47:55.995 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.993+0000 7f1ff3478700 1 --2- 172.21.15.49:0/3704495461 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f1fd802ffb0 0x7f1fd8032470 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:55.997 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.993+0000 7f1ff3478700 1 --2- 172.21.15.49:0/3704495461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fec0f47c0 0x7f1fec106700 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:55.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.993+0000 7f1ff3478700 1 -- 172.21.15.49:0/3704495461 >> 172.21.15.49:0/3704495461 conn(0x7f1fec0f02e0 msgr2=0x7f1fec0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:55.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.993+0000 7f1ff3478700 1 -- 172.21.15.49:0/3704495461 shutdown_connections 2023-11-30T21:47:55.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:55.993+0000 7f1ff3478700 1 -- 172.21.15.49:0/3704495461 wait complete. 2023-11-30T21:47:55.999 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:56.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:56 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:55.992521+0000 mon.smithi027 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.49:0/3704495461' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:47:57.376 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:47:57.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:47:57.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:57 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:55.494279+0000 mgr.smithi027.brrjcl (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:58.629 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.626+0000 7f2f4b454700 1 -- 172.21.15.49:0/3257282741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f440f78e0 msgr2=0x7f2f440f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:58.629 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.626+0000 7f2f4b454700 1 --2- 172.21.15.49:0/3257282741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f440f78e0 0x7f2f440f7d00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f2f40009fb0 tx=0x7f2f40009b30).stop 2023-11-30T21:47:58.629 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.626+0000 7f2f4b454700 1 -- 172.21.15.49:0/3257282741 shutdown_connections 2023-11-30T21:47:58.629 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.626+0000 7f2f4b454700 1 --2- 172.21.15.49:0/3257282741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f440f78e0 0x7f2f440f7d00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:58.629 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.626+0000 7f2f4b454700 1 -- 172.21.15.49:0/3257282741 >> 172.21.15.49:0/3257282741 conn(0x7f2f440f0120 msgr2=0x7f2f440f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:58.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.626+0000 7f2f4b454700 1 -- 172.21.15.49:0/3257282741 shutdown_connections 2023-11-30T21:47:58.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.626+0000 7f2f4b454700 1 -- 172.21.15.49:0/3257282741 wait complete. 2023-11-30T21:47:58.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.626+0000 7f2f4b454700 1 Processor -- start 2023-11-30T21:47:58.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.627+0000 7f2f4b454700 1 -- start start 2023-11-30T21:47:58.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.627+0000 7f2f4b454700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f440f78e0 0x7f2f440f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:58.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.627+0000 7f2f4b454700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f440f5630 con 0x7f2f440f78e0 2023-11-30T21:47:58.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.627+0000 7f2f4a452700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f440f78e0 0x7f2f440f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:58.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.627+0000 7f2f4a452700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f440f78e0 0x7f2f440f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:35556/0 (socket says 172.21.15.49:35556) 2023-11-30T21:47:58.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.627+0000 7f2f4a452700 1 -- 172.21.15.49:0/140093105 learned_addr learned my addr 172.21.15.49:0/140093105 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:47:58.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.627+0000 7f2f4a452700 1 -- 172.21.15.49:0/140093105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f40005040 con 0x7f2f440f78e0 2023-11-30T21:47:58.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.627+0000 7f2f4a452700 1 --2- 172.21.15.49:0/140093105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f440f78e0 0x7f2f440f71b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f2f40013510 tx=0x7f2f4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:58.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.628+0000 7f2f3affd700 1 -- 172.21.15.49:0/140093105 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f40009320 con 0x7f2f440f78e0 2023-11-30T21:47:58.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.628+0000 7f2f4b454700 1 -- 172.21.15.49:0/140093105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f440f57d0 con 0x7f2f440f78e0 2023-11-30T21:47:58.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.628+0000 7f2f3affd700 1 -- 172.21.15.49:0/140093105 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f40013f00 con 0x7f2f440f78e0 2023-11-30T21:47:58.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.628+0000 7f2f3affd700 1 -- 172.21.15.49:0/140093105 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f40014dc0 con 0x7f2f440f78e0 2023-11-30T21:47:58.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.628+0000 7f2f4b454700 1 -- 172.21.15.49:0/140093105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f440f5c60 con 0x7f2f440f78e0 2023-11-30T21:47:58.632 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.628+0000 7f2f3affd700 1 -- 172.21.15.49:0/140093105 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f40014420 con 0x7f2f440f78e0 2023-11-30T21:47:58.632 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.628+0000 7f2f4b454700 1 -- 172.21.15.49:0/140093105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f44059100 con 0x7f2f440f78e0 2023-11-30T21:47:58.632 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.629+0000 7f2f3affd700 1 --2- 172.21.15.49:0/140093105 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2f3002ff60 0x7f2f30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:47:58.632 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.629+0000 7f2f3affd700 1 -- 172.21.15.49:0/140093105 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f40041030 con 0x7f2f440f78e0 2023-11-30T21:47:58.632 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.629+0000 7f2f49c51700 1 --2- 172.21.15.49:0/140093105 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2f3002ff60 0x7f2f30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:47:58.632 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.629+0000 7f2f49c51700 1 --2- 172.21.15.49:0/140093105 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2f3002ff60 0x7f2f30032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f2f340096a0 tx=0x7f2f34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:47:58.634 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.633+0000 7f2f3affd700 1 -- 172.21.15.49:0/140093105 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f400095f0 con 0x7f2f440f78e0 2023-11-30T21:47:58.851 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.849+0000 7f2f4b454700 1 -- 172.21.15.49:0/140093105 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f440fbce0 con 0x7f2f440f78e0 2023-11-30T21:47:58.852 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.849+0000 7f2f3affd700 1 -- 172.21.15.49:0/140093105 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2f4001cb10 con 0x7f2f440f78e0 2023-11-30T21:47:58.852 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:47:58.852 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:47:58.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.851+0000 7f2f4b454700 1 -- 172.21.15.49:0/140093105 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2f3002ff60 msgr2=0x7f2f30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:58.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.851+0000 7f2f4b454700 1 --2- 172.21.15.49:0/140093105 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2f3002ff60 0x7f2f30032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f2f340096a0 tx=0x7f2f34006b40).stop 2023-11-30T21:47:58.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.851+0000 7f2f4b454700 1 -- 172.21.15.49:0/140093105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f440f78e0 msgr2=0x7f2f440f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:47:58.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.851+0000 7f2f4b454700 1 --2- 172.21.15.49:0/140093105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f440f78e0 0x7f2f440f71b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f2f40013510 tx=0x7f2f4000b040).stop 2023-11-30T21:47:58.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.851+0000 7f2f4b454700 1 -- 172.21.15.49:0/140093105 shutdown_connections 2023-11-30T21:47:58.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.851+0000 7f2f4b454700 1 --2- 172.21.15.49:0/140093105 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2f3002ff60 0x7f2f30032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:58.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.851+0000 7f2f4b454700 1 --2- 172.21.15.49:0/140093105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f440f78e0 0x7f2f440f71b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:47:58.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.851+0000 7f2f4b454700 1 -- 172.21.15.49:0/140093105 >> 172.21.15.49:0/140093105 conn(0x7f2f440f0120 msgr2=0x7f2f440f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:47:58.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.851+0000 7f2f4b454700 1 -- 172.21.15.49:0/140093105 shutdown_connections 2023-11-30T21:47:58.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:47:58.851+0000 7f2f4b454700 1 -- 172.21.15.49:0/140093105 wait complete. 2023-11-30T21:47:58.855 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:47:59.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:59 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:57.494548+0000 mgr.smithi027.brrjcl (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:47:59.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:47:59 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:47:58.850903+0000 mon.smithi027 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.49:0/140093105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:00.250 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:00.250 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.416+0000 7f144c06f700 1 -- 172.21.15.49:0/2002148512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14440f6510 msgr2=0x7f14440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.416+0000 7f144c06f700 1 --2- 172.21.15.49:0/2002148512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14440f6510 0x7f14440f6930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f1440009fb0 tx=0x7f1440009b30).stop 2023-11-30T21:48:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.416+0000 7f144c06f700 1 -- 172.21.15.49:0/2002148512 shutdown_connections 2023-11-30T21:48:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.416+0000 7f144c06f700 1 --2- 172.21.15.49:0/2002148512 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14440f6510 0x7f14440f6930 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.416+0000 7f144c06f700 1 -- 172.21.15.49:0/2002148512 >> 172.21.15.49:0/2002148512 conn(0x7f14440f1c70 msgr2=0x7f14440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.416+0000 7f144c06f700 1 -- 172.21.15.49:0/2002148512 shutdown_connections 2023-11-30T21:48:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.416+0000 7f144c06f700 1 -- 172.21.15.49:0/2002148512 wait complete. 2023-11-30T21:48:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.417+0000 7f144c06f700 1 Processor -- start 2023-11-30T21:48:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.417+0000 7f144c06f700 1 -- start start 2023-11-30T21:48:01.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.417+0000 7f144c06f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14440f6510 0x7f1444106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.417+0000 7f144c06f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1444106c20 con 0x7f14440f6510 2023-11-30T21:48:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.417+0000 7f1449e0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14440f6510 0x7f1444106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.418+0000 7f1449e0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14440f6510 0x7f1444106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:35576/0 (socket says 172.21.15.49:35576) 2023-11-30T21:48:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.418+0000 7f1449e0b700 1 -- 172.21.15.49:0/148556001 learned_addr learned my addr 172.21.15.49:0/148556001 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.418+0000 7f1449e0b700 1 -- 172.21.15.49:0/148556001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1440005040 con 0x7f14440f6510 2023-11-30T21:48:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.418+0000 7f1449e0b700 1 --2- 172.21.15.49:0/148556001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14440f6510 0x7f1444106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f1440013510 tx=0x7f144000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.418+0000 7f143a7fc700 1 -- 172.21.15.49:0/148556001 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1440009320 con 0x7f14440f6510 2023-11-30T21:48:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.418+0000 7f144c06f700 1 -- 172.21.15.49:0/148556001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1444106e20 con 0x7f14440f6510 2023-11-30T21:48:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.418+0000 7f143a7fc700 1 -- 172.21.15.49:0/148556001 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1440013f00 con 0x7f14440f6510 2023-11-30T21:48:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.418+0000 7f143a7fc700 1 -- 172.21.15.49:0/148556001 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1440014d90 con 0x7f14440f6510 2023-11-30T21:48:01.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.418+0000 7f144c06f700 1 -- 172.21.15.49:0/148556001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14441072b0 con 0x7f14440f6510 2023-11-30T21:48:01.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.419+0000 7f143a7fc700 1 -- 172.21.15.49:0/148556001 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14400095f0 con 0x7f14440f6510 2023-11-30T21:48:01.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.419+0000 7f144c06f700 1 -- 172.21.15.49:0/148556001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1444045ad0 con 0x7f14440f6510 2023-11-30T21:48:01.422 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.419+0000 7f143a7fc700 1 --2- 172.21.15.49:0/148556001 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f143002ff60 0x7f1430032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:01.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.419+0000 7f143a7fc700 1 -- 172.21.15.49:0/148556001 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1440041000 con 0x7f14440f6510 2023-11-30T21:48:01.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.419+0000 7f144960a700 1 --2- 172.21.15.49:0/148556001 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f143002ff60 0x7f1430032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:01.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.420+0000 7f144960a700 1 --2- 172.21.15.49:0/148556001 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f143002ff60 0x7f1430032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f14340096a0 tx=0x7f1434006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:01.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.424+0000 7f143a7fc700 1 -- 172.21.15.49:0/148556001 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1440014420 con 0x7f14440f6510 2023-11-30T21:48:01.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:01 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:47:59.494802+0000 mgr.smithi027.brrjcl (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:01.644 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.642+0000 7f144c06f700 1 -- 172.21.15.49:0/148556001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14440fbf60 con 0x7f14440f6510 2023-11-30T21:48:01.644 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.642+0000 7f143a7fc700 1 -- 172.21.15.49:0/148556001 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1440010230 con 0x7f14440f6510 2023-11-30T21:48:01.645 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:01.645 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:01.646 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.644+0000 7f144c06f700 1 -- 172.21.15.49:0/148556001 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f143002ff60 msgr2=0x7f1430032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:01.646 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.644+0000 7f144c06f700 1 --2- 172.21.15.49:0/148556001 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f143002ff60 0x7f1430032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f14340096a0 tx=0x7f1434006b40).stop 2023-11-30T21:48:01.646 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.644+0000 7f144c06f700 1 -- 172.21.15.49:0/148556001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14440f6510 msgr2=0x7f1444106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:01.646 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.644+0000 7f144c06f700 1 --2- 172.21.15.49:0/148556001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14440f6510 0x7f1444106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f1440013510 tx=0x7f144000b040).stop 2023-11-30T21:48:01.646 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.644+0000 7f144c06f700 1 -- 172.21.15.49:0/148556001 shutdown_connections 2023-11-30T21:48:01.646 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.644+0000 7f144c06f700 1 --2- 172.21.15.49:0/148556001 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f143002ff60 0x7f1430032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.644+0000 7f144c06f700 1 --2- 172.21.15.49:0/148556001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14440f6510 0x7f1444106730 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.644+0000 7f144c06f700 1 -- 172.21.15.49:0/148556001 >> 172.21.15.49:0/148556001 conn(0x7f14440f1c70 msgr2=0x7f14440f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.644+0000 7f144c06f700 1 -- 172.21.15.49:0/148556001 shutdown_connections 2023-11-30T21:48:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:01.644+0000 7f144c06f700 1 -- 172.21.15.49:0/148556001 wait complete. 2023-11-30T21:48:01.647 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:02.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:02 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:01.644005+0000 mon.smithi027 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.49:0/148556001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:03.051 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:03.051 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:03.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:03 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:01.495193+0000 mgr.smithi027.brrjcl (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:04.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.219+0000 7fedb0dd6700 1 -- 172.21.15.49:0/1509964850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0f5410 msgr2=0x7fedac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:04.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.219+0000 7fedb0dd6700 1 --2- 172.21.15.49:0/1509964850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0f5410 0x7fedac0f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fed94009fb0 tx=0x7fed94009b30).stop 2023-11-30T21:48:04.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.219+0000 7fedb0dd6700 1 -- 172.21.15.49:0/1509964850 shutdown_connections 2023-11-30T21:48:04.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.219+0000 7fedb0dd6700 1 --2- 172.21.15.49:0/1509964850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0f5410 0x7fedac0f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:04.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.219+0000 7fedb0dd6700 1 -- 172.21.15.49:0/1509964850 >> 172.21.15.49:0/1509964850 conn(0x7fedac0f0b70 msgr2=0x7fedac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:04.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.219+0000 7fedb0dd6700 1 -- 172.21.15.49:0/1509964850 shutdown_connections 2023-11-30T21:48:04.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.219+0000 7fedb0dd6700 1 -- 172.21.15.49:0/1509964850 wait complete. 2023-11-30T21:48:04.222 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.220+0000 7fedb0dd6700 1 Processor -- start 2023-11-30T21:48:04.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.220+0000 7fedb0dd6700 1 -- start start 2023-11-30T21:48:04.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.220+0000 7fedb0dd6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0f5410 0x7fedac106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:04.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.220+0000 7fedb0dd6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedac106c50 con 0x7fedac0f5410 2023-11-30T21:48:04.223 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.220+0000 7fedaa59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0f5410 0x7fedac106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:04.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.220+0000 7fedaa59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0f5410 0x7fedac106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:35588/0 (socket says 172.21.15.49:35588) 2023-11-30T21:48:04.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.220+0000 7fedaa59c700 1 -- 172.21.15.49:0/569596763 learned_addr learned my addr 172.21.15.49:0/569596763 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:04.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.221+0000 7fedaa59c700 1 -- 172.21.15.49:0/569596763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed94005040 con 0x7fedac0f5410 2023-11-30T21:48:04.225 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.221+0000 7fedaa59c700 1 --2- 172.21.15.49:0/569596763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0f5410 0x7fedac106760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fed94009f80 tx=0x7fed9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:04.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.221+0000 7feda2ffd700 1 -- 172.21.15.49:0/569596763 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed94009320 con 0x7fedac0f5410 2023-11-30T21:48:04.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.221+0000 7feda2ffd700 1 -- 172.21.15.49:0/569596763 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed94013ec0 con 0x7fedac0f5410 2023-11-30T21:48:04.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.221+0000 7feda2ffd700 1 -- 172.21.15.49:0/569596763 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed9401ece0 con 0x7fedac0f5410 2023-11-30T21:48:04.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.221+0000 7fedb0dd6700 1 -- 172.21.15.49:0/569596763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedac106e50 con 0x7fedac0f5410 2023-11-30T21:48:04.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.221+0000 7fedb0dd6700 1 -- 172.21.15.49:0/569596763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedac1072e0 con 0x7fedac0f5410 2023-11-30T21:48:04.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.222+0000 7feda2ffd700 1 -- 172.21.15.49:0/569596763 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed9401e420 con 0x7fedac0f5410 2023-11-30T21:48:04.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.222+0000 7fedb0dd6700 1 -- 172.21.15.49:0/569596763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedac045ad0 con 0x7fedac0f5410 2023-11-30T21:48:04.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.222+0000 7feda2ffd700 1 --2- 172.21.15.49:0/569596763 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fed9802ffb0 0x7fed98032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:04.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.222+0000 7feda2ffd700 1 -- 172.21.15.49:0/569596763 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed94040cb0 con 0x7fedac0f5410 2023-11-30T21:48:04.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.222+0000 7feda9d9b700 1 --2- 172.21.15.49:0/569596763 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fed9802ffb0 0x7fed98032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:04.226 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.223+0000 7feda9d9b700 1 --2- 172.21.15.49:0/569596763 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fed9802ffb0 0x7fed98032470 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fed9c0096a0 tx=0x7fed9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:04.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.227+0000 7feda2ffd700 1 -- 172.21.15.49:0/569596763 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed9401d380 con 0x7fedac0f5410 2023-11-30T21:48:04.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.447+0000 7fedb0dd6700 1 -- 172.21.15.49:0/569596763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fedac108190 con 0x7fedac0f5410 2023-11-30T21:48:04.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.449+0000 7feda2ffd700 1 -- 172.21.15.49:0/569596763 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fed9402c2d0 con 0x7fedac0f5410 2023-11-30T21:48:04.454 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:04.454 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:04.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.449+0000 7fedb0dd6700 1 -- 172.21.15.49:0/569596763 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fed9802ffb0 msgr2=0x7fed98032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:04.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.449+0000 7fedb0dd6700 1 --2- 172.21.15.49:0/569596763 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fed9802ffb0 0x7fed98032470 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fed9c0096a0 tx=0x7fed9c006b40).stop 2023-11-30T21:48:04.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.449+0000 7fedb0dd6700 1 -- 172.21.15.49:0/569596763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0f5410 msgr2=0x7fedac106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:04.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.449+0000 7fedb0dd6700 1 --2- 172.21.15.49:0/569596763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0f5410 0x7fedac106760 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fed94009f80 tx=0x7fed9400b040).stop 2023-11-30T21:48:04.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.450+0000 7fedb0dd6700 1 -- 172.21.15.49:0/569596763 shutdown_connections 2023-11-30T21:48:04.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.450+0000 7fedb0dd6700 1 --2- 172.21.15.49:0/569596763 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fed9802ffb0 0x7fed98032470 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:04.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.450+0000 7fedb0dd6700 1 --2- 172.21.15.49:0/569596763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedac0f5410 0x7fedac106760 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:04.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.450+0000 7fedb0dd6700 1 -- 172.21.15.49:0/569596763 >> 172.21.15.49:0/569596763 conn(0x7fedac0f0b70 msgr2=0x7fedac05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:04.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.450+0000 7fedb0dd6700 1 -- 172.21.15.49:0/569596763 shutdown_connections 2023-11-30T21:48:04.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:04.450+0000 7fedb0dd6700 1 -- 172.21.15.49:0/569596763 wait complete. 2023-11-30T21:48:04.456 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:03.586768+0000 mon.smithi027 (mon.0) 192 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/mirror_snapshot_schedule"}]: dispatch 2023-11-30T21:48:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:03.593736+0000 mon.smithi027 (mon.0) 193 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/trash_purge_schedule"}]: dispatch 2023-11-30T21:48:05.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:03.495525+0000 mgr.smithi027.brrjcl (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:05.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:04.449349+0000 mon.smithi027 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.49:0/569596763' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:05.845 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:05.845 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:07.058 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.057+0000 7f723e0e8700 1 -- 172.21.15.49:0/2750901863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7238060280 msgr2=0x7f72380f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:07.059 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.057+0000 7f723e0e8700 1 --2- 172.21.15.49:0/2750901863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7238060280 0x7f72380f90f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f7228009fb0 tx=0x7f7228009b30).stop 2023-11-30T21:48:07.059 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.058+0000 7f723e0e8700 1 -- 172.21.15.49:0/2750901863 shutdown_connections 2023-11-30T21:48:07.059 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.058+0000 7f723e0e8700 1 --2- 172.21.15.49:0/2750901863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7238060280 0x7f72380f90f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:07.059 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.058+0000 7f723e0e8700 1 -- 172.21.15.49:0/2750901863 >> 172.21.15.49:0/2750901863 conn(0x7f72380f0b70 msgr2=0x7f72380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:07.059 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.058+0000 7f723e0e8700 1 -- 172.21.15.49:0/2750901863 shutdown_connections 2023-11-30T21:48:07.059 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.058+0000 7f723e0e8700 1 -- 172.21.15.49:0/2750901863 wait complete. 2023-11-30T21:48:07.059 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.058+0000 7f723e0e8700 1 Processor -- start 2023-11-30T21:48:07.060 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.058+0000 7f723e0e8700 1 -- start start 2023-11-30T21:48:07.060 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.059+0000 7f723e0e8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7238060280 0x7f72381066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:07.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.059+0000 7f723e0e8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7238106bd0 con 0x7f7238060280 2023-11-30T21:48:07.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.059+0000 7f72377fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7238060280 0x7f72381066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:07.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.059+0000 7f72377fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7238060280 0x7f72381066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:35610/0 (socket says 172.21.15.49:35610) 2023-11-30T21:48:07.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.059+0000 7f72377fe700 1 -- 172.21.15.49:0/1018314376 learned_addr learned my addr 172.21.15.49:0/1018314376 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:07.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.059+0000 7f72377fe700 1 -- 172.21.15.49:0/1018314376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7228005040 con 0x7f7238060280 2023-11-30T21:48:07.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.060+0000 7f72377fe700 1 --2- 172.21.15.49:0/1018314376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7238060280 0x7f72381066e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f7228013510 tx=0x7f722800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:07.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.060+0000 7f721ffff700 1 -- 172.21.15.49:0/1018314376 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7228009320 con 0x7f7238060280 2023-11-30T21:48:07.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.060+0000 7f723e0e8700 1 -- 172.21.15.49:0/1018314376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7238106dd0 con 0x7f7238060280 2023-11-30T21:48:07.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.060+0000 7f721ffff700 1 -- 172.21.15.49:0/1018314376 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7228013f00 con 0x7f7238060280 2023-11-30T21:48:07.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.060+0000 7f721ffff700 1 -- 172.21.15.49:0/1018314376 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7228014dc0 con 0x7f7238060280 2023-11-30T21:48:07.061 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.060+0000 7f723e0e8700 1 -- 172.21.15.49:0/1018314376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7238107260 con 0x7f7238060280 2023-11-30T21:48:07.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.061+0000 7f721ffff700 1 -- 172.21.15.49:0/1018314376 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f722801d3d0 con 0x7f7238060280 2023-11-30T21:48:07.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.061+0000 7f723e0e8700 1 -- 172.21.15.49:0/1018314376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7238045ad0 con 0x7f7238060280 2023-11-30T21:48:07.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.061+0000 7f721ffff700 1 --2- 172.21.15.49:0/1018314376 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f722002ff60 0x7f7220032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:07.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.061+0000 7f721ffff700 1 -- 172.21.15.49:0/1018314376 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72280413f0 con 0x7f7238060280 2023-11-30T21:48:07.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.061+0000 7f7236ffd700 1 --2- 172.21.15.49:0/1018314376 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f722002ff60 0x7f7220032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:07.063 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.061+0000 7f7236ffd700 1 --2- 172.21.15.49:0/1018314376 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f722002ff60 0x7f7220032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f722c0096a0 tx=0x7f722c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:07.066 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.065+0000 7f721ffff700 1 -- 172.21.15.49:0/1018314376 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7228014600 con 0x7f7238060280 2023-11-30T21:48:07.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.284+0000 7f723e0e8700 1 -- 172.21.15.49:0/1018314376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7238108110 con 0x7f7238060280 2023-11-30T21:48:07.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.285+0000 7f721ffff700 1 -- 172.21.15.49:0/1018314376 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7228014420 con 0x7f7238060280 2023-11-30T21:48:07.286 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:07.286 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:07.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.286+0000 7f723e0e8700 1 -- 172.21.15.49:0/1018314376 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f722002ff60 msgr2=0x7f7220032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:07.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.286+0000 7f723e0e8700 1 --2- 172.21.15.49:0/1018314376 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f722002ff60 0x7f7220032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f722c0096a0 tx=0x7f722c006b40).stop 2023-11-30T21:48:07.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.286+0000 7f723e0e8700 1 -- 172.21.15.49:0/1018314376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7238060280 msgr2=0x7f72381066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:07.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.286+0000 7f723e0e8700 1 --2- 172.21.15.49:0/1018314376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7238060280 0x7f72381066e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f7228013510 tx=0x7f722800b040).stop 2023-11-30T21:48:07.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.286+0000 7f723e0e8700 1 -- 172.21.15.49:0/1018314376 shutdown_connections 2023-11-30T21:48:07.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.286+0000 7f723e0e8700 1 --2- 172.21.15.49:0/1018314376 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f722002ff60 0x7f7220032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:07.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.286+0000 7f723e0e8700 1 --2- 172.21.15.49:0/1018314376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7238060280 0x7f72381066e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:07.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.286+0000 7f723e0e8700 1 -- 172.21.15.49:0/1018314376 >> 172.21.15.49:0/1018314376 conn(0x7f72380f0b70 msgr2=0x7f72380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:07.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.287+0000 7f723e0e8700 1 -- 172.21.15.49:0/1018314376 shutdown_connections 2023-11-30T21:48:07.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:07.287+0000 7f723e0e8700 1 -- 172.21.15.49:0/1018314376 wait complete. 2023-11-30T21:48:07.289 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:07.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:07 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:05.495739+0000 mgr.smithi027.brrjcl (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:08.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:08 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:07.285428+0000 mon.smithi027 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.49:0/1018314376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:08.705 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:08.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:09.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:09 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:07.496083+0000 mgr.smithi027.brrjcl (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:09.939 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.937+0000 7f13e5b76700 1 -- 172.21.15.49:0/2079767184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00601c0 msgr2=0x7f13e00f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:09.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.937+0000 7f13e5b76700 1 --2- 172.21.15.49:0/2079767184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00601c0 0x7f13e00f91b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f13d0009fb0 tx=0x7f13d0009b30).stop 2023-11-30T21:48:09.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.937+0000 7f13e5b76700 1 -- 172.21.15.49:0/2079767184 shutdown_connections 2023-11-30T21:48:09.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.937+0000 7f13e5b76700 1 --2- 172.21.15.49:0/2079767184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00601c0 0x7f13e00f91b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:09.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.937+0000 7f13e5b76700 1 -- 172.21.15.49:0/2079767184 >> 172.21.15.49:0/2079767184 conn(0x7f13e00f0b70 msgr2=0x7f13e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:09.942 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.937+0000 7f13e5b76700 1 -- 172.21.15.49:0/2079767184 shutdown_connections 2023-11-30T21:48:09.942 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.937+0000 7f13e5b76700 1 -- 172.21.15.49:0/2079767184 wait complete. 2023-11-30T21:48:09.942 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.937+0000 7f13e5b76700 1 Processor -- start 2023-11-30T21:48:09.942 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.938+0000 7f13e5b76700 1 -- start start 2023-11-30T21:48:09.943 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.938+0000 7f13e5b76700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00601c0 0x7f13e01066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:09.943 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.938+0000 7f13e5b76700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e0106bc0 con 0x7f13e00601c0 2023-11-30T21:48:09.943 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.938+0000 7f13df7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00601c0 0x7f13e01066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:09.944 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.938+0000 7f13df7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00601c0 0x7f13e01066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:53442/0 (socket says 172.21.15.49:53442) 2023-11-30T21:48:09.944 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.938+0000 7f13df7fe700 1 -- 172.21.15.49:0/2619343835 learned_addr learned my addr 172.21.15.49:0/2619343835 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:09.944 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.938+0000 7f13df7fe700 1 -- 172.21.15.49:0/2619343835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d0005040 con 0x7f13e00601c0 2023-11-30T21:48:09.944 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.938+0000 7f13df7fe700 1 --2- 172.21.15.49:0/2619343835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00601c0 0x7f13e01066d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f13d0009830 tx=0x7f13d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:09.947 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.939+0000 7f13d7fff700 1 -- 172.21.15.49:0/2619343835 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13d0009320 con 0x7f13e00601c0 2023-11-30T21:48:09.947 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.939+0000 7f13e5b76700 1 -- 172.21.15.49:0/2619343835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e0106dc0 con 0x7f13e00601c0 2023-11-30T21:48:09.947 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.939+0000 7f13d7fff700 1 -- 172.21.15.49:0/2619343835 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13d00136c0 con 0x7f13e00601c0 2023-11-30T21:48:09.947 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.939+0000 7f13d7fff700 1 -- 172.21.15.49:0/2619343835 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13d001ee90 con 0x7f13e00601c0 2023-11-30T21:48:09.948 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.939+0000 7f13e5b76700 1 -- 172.21.15.49:0/2619343835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e0107250 con 0x7f13e00601c0 2023-11-30T21:48:09.948 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.939+0000 7f13d7fff700 1 -- 172.21.15.49:0/2619343835 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13d001e420 con 0x7f13e00601c0 2023-11-30T21:48:09.948 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.940+0000 7f13e5b76700 1 -- 172.21.15.49:0/2619343835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e0045ad0 con 0x7f13e00601c0 2023-11-30T21:48:09.948 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.940+0000 7f13d7fff700 1 --2- 172.21.15.49:0/2619343835 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f13cc02ff10 0x7f13cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:09.949 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.940+0000 7f13d7fff700 1 -- 172.21.15.49:0/2619343835 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f13d0041080 con 0x7f13e00601c0 2023-11-30T21:48:09.949 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.940+0000 7f13deffd700 1 --2- 172.21.15.49:0/2619343835 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f13cc02ff10 0x7f13cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:09.950 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.940+0000 7f13deffd700 1 --2- 172.21.15.49:0/2619343835 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f13cc02ff10 0x7f13cc0323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f13c80096a0 tx=0x7f13c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:09.950 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:09.944+0000 7f13d7fff700 1 -- 172.21.15.49:0/2619343835 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13d006d2b0 con 0x7f13e00601c0 2023-11-30T21:48:10.167 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:10.166+0000 7f13e5b76700 1 -- 172.21.15.49:0/2619343835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13e0108100 con 0x7f13e00601c0 2023-11-30T21:48:10.167 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:10.166+0000 7f13d7fff700 1 -- 172.21.15.49:0/2619343835 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f13d002c2d0 con 0x7f13e00601c0 2023-11-30T21:48:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:10.168 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:10.169 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:10.168+0000 7f13e5b76700 1 -- 172.21.15.49:0/2619343835 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f13cc02ff10 msgr2=0x7f13cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:10.169 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:10.168+0000 7f13e5b76700 1 --2- 172.21.15.49:0/2619343835 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f13cc02ff10 0x7f13cc0323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f13c80096a0 tx=0x7f13c8006b40).stop 2023-11-30T21:48:10.169 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:10.168+0000 7f13e5b76700 1 -- 172.21.15.49:0/2619343835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00601c0 msgr2=0x7f13e01066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:10.169 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:10.168+0000 7f13e5b76700 1 --2- 172.21.15.49:0/2619343835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00601c0 0x7f13e01066d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f13d0009830 tx=0x7f13d000b040).stop 2023-11-30T21:48:10.170 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:10.168+0000 7f13e5b76700 1 -- 172.21.15.49:0/2619343835 shutdown_connections 2023-11-30T21:48:10.170 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:10.168+0000 7f13e5b76700 1 --2- 172.21.15.49:0/2619343835 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f13cc02ff10 0x7f13cc0323d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:10.170 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:10.168+0000 7f13e5b76700 1 --2- 172.21.15.49:0/2619343835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00601c0 0x7f13e01066d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:10.170 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:10.168+0000 7f13e5b76700 1 -- 172.21.15.49:0/2619343835 >> 172.21.15.49:0/2619343835 conn(0x7f13e00f0b70 msgr2=0x7f13e00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:10.170 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:10.168+0000 7f13e5b76700 1 -- 172.21.15.49:0/2619343835 shutdown_connections 2023-11-30T21:48:10.170 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:10.168+0000 7f13e5b76700 1 -- 172.21.15.49:0/2619343835 wait complete. 2023-11-30T21:48:10.171 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:10.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:10 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:10.166945+0000 mon.smithi027 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.49:0/2619343835' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:11.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:09.496453+0000 mgr.smithi027.brrjcl (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:11.593 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:11.594 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.819+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/3194812963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c140f5410 msgr2=0x7f0c140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.819+0000 7f0c1a2c4700 1 --2- 172.21.15.49:0/3194812963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c140f5410 0x7f0c140f5830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f0c04009fb0 tx=0x7f0c04009b30).stop 2023-11-30T21:48:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.819+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/3194812963 shutdown_connections 2023-11-30T21:48:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.819+0000 7f0c1a2c4700 1 --2- 172.21.15.49:0/3194812963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c140f5410 0x7f0c140f5830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.819+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/3194812963 >> 172.21.15.49:0/3194812963 conn(0x7f0c140f0b70 msgr2=0x7f0c140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.819+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/3194812963 shutdown_connections 2023-11-30T21:48:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.819+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/3194812963 wait complete. 2023-11-30T21:48:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.819+0000 7f0c1a2c4700 1 Processor -- start 2023-11-30T21:48:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.820+0000 7f0c1a2c4700 1 -- start start 2023-11-30T21:48:12.823 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.820+0000 7f0c1a2c4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c140f5410 0x7f0c14108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:12.823 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.820+0000 7f0c1a2c4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c14108e00 con 0x7f0c140f5410 2023-11-30T21:48:12.823 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.820+0000 7f0c13fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c140f5410 0x7f0c14108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:12.823 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.820+0000 7f0c13fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c140f5410 0x7f0c14108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:53464/0 (socket says 172.21.15.49:53464) 2023-11-30T21:48:12.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.820+0000 7f0c13fff700 1 -- 172.21.15.49:0/2658718939 learned_addr learned my addr 172.21.15.49:0/2658718939 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:12.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.820+0000 7f0c13fff700 1 -- 172.21.15.49:0/2658718939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c04005040 con 0x7f0c140f5410 2023-11-30T21:48:12.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.821+0000 7f0c13fff700 1 --2- 172.21.15.49:0/2658718939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c140f5410 0x7f0c14108910 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f0c04013510 tx=0x7f0c0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:12.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.821+0000 7f0c10ff9700 1 -- 172.21.15.49:0/2658718939 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c04009320 con 0x7f0c140f5410 2023-11-30T21:48:12.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.821+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/2658718939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c14109000 con 0x7f0c140f5410 2023-11-30T21:48:12.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.821+0000 7f0c10ff9700 1 -- 172.21.15.49:0/2658718939 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c04013f00 con 0x7f0c140f5410 2023-11-30T21:48:12.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.821+0000 7f0c10ff9700 1 -- 172.21.15.49:0/2658718939 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c04014d90 con 0x7f0c140f5410 2023-11-30T21:48:12.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.821+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/2658718939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c14109490 con 0x7f0c140f5410 2023-11-30T21:48:12.825 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.822+0000 7f0c10ff9700 1 -- 172.21.15.49:0/2658718939 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c04014420 con 0x7f0c140f5410 2023-11-30T21:48:12.825 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.822+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/2658718939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c14045ad0 con 0x7f0c140f5410 2023-11-30T21:48:12.825 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.822+0000 7f0c10ff9700 1 --2- 172.21.15.49:0/2658718939 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0c0002ff60 0x7f0c00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:12.825 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.822+0000 7f0c10ff9700 1 -- 172.21.15.49:0/2658718939 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c04041000 con 0x7f0c140f5410 2023-11-30T21:48:12.825 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.822+0000 7f0c137fe700 1 --2- 172.21.15.49:0/2658718939 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0c0002ff60 0x7f0c00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:12.825 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.822+0000 7f0c137fe700 1 --2- 172.21.15.49:0/2658718939 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0c0002ff60 0x7f0c00032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f0bfc0096a0 tx=0x7f0bfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:12.827 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:12.826+0000 7f0c10ff9700 1 -- 172.21.15.49:0/2658718939 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c040095f0 con 0x7f0c140f5410 2023-11-30T21:48:13.046 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:13.045+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/2658718939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c140fae60 con 0x7f0c140f5410 2023-11-30T21:48:13.046 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:13.046+0000 7f0c10ff9700 1 -- 172.21.15.49:0/2658718939 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0c0401db70 con 0x7f0c140f5410 2023-11-30T21:48:13.047 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:13.047 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:13.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:13.047+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/2658718939 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0c0002ff60 msgr2=0x7f0c00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:13.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:13.047+0000 7f0c1a2c4700 1 --2- 172.21.15.49:0/2658718939 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0c0002ff60 0x7f0c00032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f0bfc0096a0 tx=0x7f0bfc006b40).stop 2023-11-30T21:48:13.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:13.047+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/2658718939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c140f5410 msgr2=0x7f0c14108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:13.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:13.047+0000 7f0c1a2c4700 1 --2- 172.21.15.49:0/2658718939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c140f5410 0x7f0c14108910 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f0c04013510 tx=0x7f0c0400b040).stop 2023-11-30T21:48:13.048 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:13.047+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/2658718939 shutdown_connections 2023-11-30T21:48:13.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:13.048+0000 7f0c1a2c4700 1 --2- 172.21.15.49:0/2658718939 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0c0002ff60 0x7f0c00032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:13.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:13.048+0000 7f0c1a2c4700 1 --2- 172.21.15.49:0/2658718939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c140f5410 0x7f0c14108910 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:13.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:13.048+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/2658718939 >> 172.21.15.49:0/2658718939 conn(0x7f0c140f0b70 msgr2=0x7f0c1405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:13.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:13.048+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/2658718939 shutdown_connections 2023-11-30T21:48:13.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:13.048+0000 7f0c1a2c4700 1 -- 172.21.15.49:0/2658718939 wait complete. 2023-11-30T21:48:13.049 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:13.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:13 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:11.496829+0000 mgr.smithi027.brrjcl (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:13.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:13 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:13.046339+0000 mon.smithi027 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.49:0/2658718939' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:14.459 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:14.460 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:15.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:15 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:13.497289+0000 mgr.smithi027.brrjcl (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:15.676 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.675+0000 7fd6aef35700 1 -- 172.21.15.49:0/3952036676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a8060280 msgr2=0x7fd6a80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:15.676 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.675+0000 7fd6aef35700 1 --2- 172.21.15.49:0/3952036676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a8060280 0x7fd6a80f90f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fd69c009fb0 tx=0x7fd69c009b30).stop 2023-11-30T21:48:15.676 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.675+0000 7fd6aef35700 1 -- 172.21.15.49:0/3952036676 shutdown_connections 2023-11-30T21:48:15.676 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.675+0000 7fd6aef35700 1 --2- 172.21.15.49:0/3952036676 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a8060280 0x7fd6a80f90f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:15.676 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.675+0000 7fd6aef35700 1 -- 172.21.15.49:0/3952036676 >> 172.21.15.49:0/3952036676 conn(0x7fd6a80f0b70 msgr2=0x7fd6a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:15.677 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.675+0000 7fd6aef35700 1 -- 172.21.15.49:0/3952036676 shutdown_connections 2023-11-30T21:48:15.677 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.675+0000 7fd6aef35700 1 -- 172.21.15.49:0/3952036676 wait complete. 2023-11-30T21:48:15.677 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.676+0000 7fd6aef35700 1 Processor -- start 2023-11-30T21:48:15.677 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.676+0000 7fd6aef35700 1 -- start start 2023-11-30T21:48:15.678 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.676+0000 7fd6aef35700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a8060280 0x7fd6a81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:15.678 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.676+0000 7fd6aef35700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a8106bd0 con 0x7fd6a8060280 2023-11-30T21:48:15.679 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.676+0000 7fd6accd1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a8060280 0x7fd6a81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:15.679 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.676+0000 7fd6accd1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a8060280 0x7fd6a81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:53480/0 (socket says 172.21.15.49:53480) 2023-11-30T21:48:15.679 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.676+0000 7fd6accd1700 1 -- 172.21.15.49:0/1875068597 learned_addr learned my addr 172.21.15.49:0/1875068597 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:15.679 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.676+0000 7fd6accd1700 1 -- 172.21.15.49:0/1875068597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c005040 con 0x7fd6a8060280 2023-11-30T21:48:15.679 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.677+0000 7fd6accd1700 1 --2- 172.21.15.49:0/1875068597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a8060280 0x7fd6a81066e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fd69c013510 tx=0x7fd69c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:15.679 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.677+0000 7fd6a57fa700 1 -- 172.21.15.49:0/1875068597 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd69c009320 con 0x7fd6a8060280 2023-11-30T21:48:15.679 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.677+0000 7fd6aef35700 1 -- 172.21.15.49:0/1875068597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a8106dd0 con 0x7fd6a8060280 2023-11-30T21:48:15.679 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.677+0000 7fd6a57fa700 1 -- 172.21.15.49:0/1875068597 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd69c013f00 con 0x7fd6a8060280 2023-11-30T21:48:15.679 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.677+0000 7fd6a57fa700 1 -- 172.21.15.49:0/1875068597 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd69c014dc0 con 0x7fd6a8060280 2023-11-30T21:48:15.679 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.677+0000 7fd6aef35700 1 -- 172.21.15.49:0/1875068597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a8107260 con 0x7fd6a8060280 2023-11-30T21:48:15.679 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.678+0000 7fd6a57fa700 1 -- 172.21.15.49:0/1875068597 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd69c01d3d0 con 0x7fd6a8060280 2023-11-30T21:48:15.679 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.678+0000 7fd6a57fa700 1 --2- 172.21.15.49:0/1875068597 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd69802ff60 0x7fd698032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:15.680 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.678+0000 7fd6a57fa700 1 -- 172.21.15.49:0/1875068597 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd69c041040 con 0x7fd6a8060280 2023-11-30T21:48:15.680 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.678+0000 7fd6aef35700 1 -- 172.21.15.49:0/1875068597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a8045ad0 con 0x7fd6a8060280 2023-11-30T21:48:15.680 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.678+0000 7fd6a7fff700 1 --2- 172.21.15.49:0/1875068597 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd69802ff60 0x7fd698032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:15.680 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.678+0000 7fd6a7fff700 1 --2- 172.21.15.49:0/1875068597 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd69802ff60 0x7fd698032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fd6940096a0 tx=0x7fd694006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:15.683 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.683+0000 7fd6a57fa700 1 -- 172.21.15.49:0/1875068597 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd69c012920 con 0x7fd6a8060280 2023-11-30T21:48:15.900 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.899+0000 7fd6aef35700 1 -- 172.21.15.49:0/1875068597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6a80fbf50 con 0x7fd6a8060280 2023-11-30T21:48:15.901 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.900+0000 7fd6a57fa700 1 -- 172.21.15.49:0/1875068597 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd69c012330 con 0x7fd6a8060280 2023-11-30T21:48:15.901 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:15.901 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:15.903 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.902+0000 7fd6aef35700 1 -- 172.21.15.49:0/1875068597 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd69802ff60 msgr2=0x7fd698032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:15.903 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.902+0000 7fd6aef35700 1 --2- 172.21.15.49:0/1875068597 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd69802ff60 0x7fd698032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fd6940096a0 tx=0x7fd694006b40).stop 2023-11-30T21:48:15.903 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.902+0000 7fd6aef35700 1 -- 172.21.15.49:0/1875068597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a8060280 msgr2=0x7fd6a81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:15.903 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.902+0000 7fd6aef35700 1 --2- 172.21.15.49:0/1875068597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a8060280 0x7fd6a81066e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fd69c013510 tx=0x7fd69c00b040).stop 2023-11-30T21:48:15.903 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.902+0000 7fd6aef35700 1 -- 172.21.15.49:0/1875068597 shutdown_connections 2023-11-30T21:48:15.903 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.902+0000 7fd6aef35700 1 --2- 172.21.15.49:0/1875068597 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd69802ff60 0x7fd698032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:15.904 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.902+0000 7fd6aef35700 1 --2- 172.21.15.49:0/1875068597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6a8060280 0x7fd6a81066e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:15.904 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.902+0000 7fd6aef35700 1 -- 172.21.15.49:0/1875068597 >> 172.21.15.49:0/1875068597 conn(0x7fd6a80f0b70 msgr2=0x7fd6a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:15.904 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.902+0000 7fd6aef35700 1 -- 172.21.15.49:0/1875068597 shutdown_connections 2023-11-30T21:48:15.904 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:15.902+0000 7fd6aef35700 1 -- 172.21.15.49:0/1875068597 wait complete. 2023-11-30T21:48:15.904 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:16.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:16 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:15.900548+0000 mon.smithi027 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.49:0/1875068597' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:17.337 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:17.337 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:17.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:15.497603+0000 mgr.smithi027.brrjcl (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:18.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.541+0000 7f66dff50700 1 -- 172.21.15.49:0/3585910957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66d80f5410 msgr2=0x7f66d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:18.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.541+0000 7f66dff50700 1 --2- 172.21.15.49:0/3585910957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66d80f5410 0x7f66d80f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f66d4009fb0 tx=0x7f66d4009b30).stop 2023-11-30T21:48:18.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.541+0000 7f66dff50700 1 -- 172.21.15.49:0/3585910957 shutdown_connections 2023-11-30T21:48:18.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.541+0000 7f66dff50700 1 --2- 172.21.15.49:0/3585910957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66d80f5410 0x7f66d80f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:18.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.541+0000 7f66dff50700 1 -- 172.21.15.49:0/3585910957 >> 172.21.15.49:0/3585910957 conn(0x7f66d80f0b70 msgr2=0x7f66d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:18.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.541+0000 7f66dff50700 1 -- 172.21.15.49:0/3585910957 shutdown_connections 2023-11-30T21:48:18.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.541+0000 7f66dff50700 1 -- 172.21.15.49:0/3585910957 wait complete. 2023-11-30T21:48:18.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.541+0000 7f66dff50700 1 Processor -- start 2023-11-30T21:48:18.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.541+0000 7f66dff50700 1 -- start start 2023-11-30T21:48:18.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.541+0000 7f66dff50700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66d80f5410 0x7f66d8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:18.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.541+0000 7f66dff50700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d8105b20 con 0x7f66d80f5410 2023-11-30T21:48:18.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.542+0000 7f66ddcec700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66d80f5410 0x7f66d8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:18.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.542+0000 7f66ddcec700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66d80f5410 0x7f66d8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:34240/0 (socket says 172.21.15.49:34240) 2023-11-30T21:48:18.549 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.542+0000 7f66ddcec700 1 -- 172.21.15.49:0/4159675370 learned_addr learned my addr 172.21.15.49:0/4159675370 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:18.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.542+0000 7f66ddcec700 1 -- 172.21.15.49:0/4159675370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d4005040 con 0x7f66d80f5410 2023-11-30T21:48:18.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.542+0000 7f66ddcec700 1 --2- 172.21.15.49:0/4159675370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66d80f5410 0x7f66d8105630 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f66d4013510 tx=0x7f66d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:18.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.543+0000 7f66ca7fc700 1 -- 172.21.15.49:0/4159675370 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66d4009320 con 0x7f66d80f5410 2023-11-30T21:48:18.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.543+0000 7f66dff50700 1 -- 172.21.15.49:0/4159675370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d8105d20 con 0x7f66d80f5410 2023-11-30T21:48:18.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.543+0000 7f66ca7fc700 1 -- 172.21.15.49:0/4159675370 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66d4013f00 con 0x7f66d80f5410 2023-11-30T21:48:18.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.543+0000 7f66ca7fc700 1 -- 172.21.15.49:0/4159675370 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66d4014d90 con 0x7f66d80f5410 2023-11-30T21:48:18.551 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.543+0000 7f66dff50700 1 -- 172.21.15.49:0/4159675370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d81061b0 con 0x7f66d80f5410 2023-11-30T21:48:18.551 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.543+0000 7f66ca7fc700 1 -- 172.21.15.49:0/4159675370 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66d4014420 con 0x7f66d80f5410 2023-11-30T21:48:18.551 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.543+0000 7f66dff50700 1 -- 172.21.15.49:0/4159675370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d8045ad0 con 0x7f66d80f5410 2023-11-30T21:48:18.552 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.544+0000 7f66ca7fc700 1 --2- 172.21.15.49:0/4159675370 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f66c402ff60 0x7f66c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:18.552 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.544+0000 7f66ca7fc700 1 -- 172.21.15.49:0/4159675370 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66d4041000 con 0x7f66d80f5410 2023-11-30T21:48:18.552 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.544+0000 7f66dd4eb700 1 --2- 172.21.15.49:0/4159675370 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f66c402ff60 0x7f66c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:18.552 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.544+0000 7f66dd4eb700 1 --2- 172.21.15.49:0/4159675370 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f66c402ff60 0x7f66c4032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f66cc0096a0 tx=0x7f66cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:18.553 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.548+0000 7f66ca7fc700 1 -- 172.21.15.49:0/4159675370 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66d40095f0 con 0x7f66d80f5410 2023-11-30T21:48:18.770 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.769+0000 7f66dff50700 1 -- 172.21.15.49:0/4159675370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66d80fae60 con 0x7f66d80f5410 2023-11-30T21:48:18.770 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.770+0000 7f66ca7fc700 1 -- 172.21.15.49:0/4159675370 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f66d401db70 con 0x7f66d80f5410 2023-11-30T21:48:18.771 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:18.771 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:18.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.772+0000 7f66dff50700 1 -- 172.21.15.49:0/4159675370 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f66c402ff60 msgr2=0x7f66c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:18.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.772+0000 7f66dff50700 1 --2- 172.21.15.49:0/4159675370 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f66c402ff60 0x7f66c4032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f66cc0096a0 tx=0x7f66cc006b40).stop 2023-11-30T21:48:18.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.772+0000 7f66dff50700 1 -- 172.21.15.49:0/4159675370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66d80f5410 msgr2=0x7f66d8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:18.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.772+0000 7f66dff50700 1 --2- 172.21.15.49:0/4159675370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66d80f5410 0x7f66d8105630 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f66d4013510 tx=0x7f66d400b040).stop 2023-11-30T21:48:18.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.772+0000 7f66dff50700 1 -- 172.21.15.49:0/4159675370 shutdown_connections 2023-11-30T21:48:18.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.772+0000 7f66dff50700 1 --2- 172.21.15.49:0/4159675370 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f66c402ff60 0x7f66c4032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:18.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.772+0000 7f66dff50700 1 --2- 172.21.15.49:0/4159675370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66d80f5410 0x7f66d8105630 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:18.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.772+0000 7f66dff50700 1 -- 172.21.15.49:0/4159675370 >> 172.21.15.49:0/4159675370 conn(0x7f66d80f0b70 msgr2=0x7f66d80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:18.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.772+0000 7f66dff50700 1 -- 172.21.15.49:0/4159675370 shutdown_connections 2023-11-30T21:48:18.773 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:18.772+0000 7f66dff50700 1 -- 172.21.15.49:0/4159675370 wait complete. 2023-11-30T21:48:18.773 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:19.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:17.497957+0000 mgr.smithi027.brrjcl (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:19.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:18.770670+0000 mon.smithi027 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.49:0/4159675370' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:20.226 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:20.302 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:21.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:21 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:19.498278+0000 mgr.smithi027.brrjcl (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:21.553 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.551+0000 7fe1926c5700 1 -- 172.21.15.49:0/562868114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe18c0f6510 msgr2=0x7fe18c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:21.553 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.551+0000 7fe1926c5700 1 --2- 172.21.15.49:0/562868114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe18c0f6510 0x7fe18c0f6930 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fe17c009fb0 tx=0x7fe17c009b30).stop 2023-11-30T21:48:21.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.552+0000 7fe1926c5700 1 -- 172.21.15.49:0/562868114 shutdown_connections 2023-11-30T21:48:21.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.552+0000 7fe1926c5700 1 --2- 172.21.15.49:0/562868114 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe18c0f6510 0x7fe18c0f6930 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:21.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.552+0000 7fe1926c5700 1 -- 172.21.15.49:0/562868114 >> 172.21.15.49:0/562868114 conn(0x7fe18c0f1c70 msgr2=0x7fe18c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:21.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.552+0000 7fe1926c5700 1 -- 172.21.15.49:0/562868114 shutdown_connections 2023-11-30T21:48:21.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.552+0000 7fe1926c5700 1 -- 172.21.15.49:0/562868114 wait complete. 2023-11-30T21:48:21.555 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.552+0000 7fe1926c5700 1 Processor -- start 2023-11-30T21:48:21.555 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.553+0000 7fe1926c5700 1 -- start start 2023-11-30T21:48:21.555 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.553+0000 7fe1926c5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe18c0f6510 0x7fe18c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:21.555 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.553+0000 7fe1926c5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c106c20 con 0x7fe18c0f6510 2023-11-30T21:48:21.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.553+0000 7fe18bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe18c0f6510 0x7fe18c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:21.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.553+0000 7fe18bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe18c0f6510 0x7fe18c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:34262/0 (socket says 172.21.15.49:34262) 2023-11-30T21:48:21.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.553+0000 7fe18bfff700 1 -- 172.21.15.49:0/2311180904 learned_addr learned my addr 172.21.15.49:0/2311180904 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:21.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.553+0000 7fe18bfff700 1 -- 172.21.15.49:0/2311180904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c005040 con 0x7fe18c0f6510 2023-11-30T21:48:21.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.554+0000 7fe18bfff700 1 --2- 172.21.15.49:0/2311180904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe18c0f6510 0x7fe18c106730 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fe17c014000 tx=0x7fe17c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:21.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.554+0000 7fe188ff9700 1 -- 172.21.15.49:0/2311180904 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe17c0095f0 con 0x7fe18c0f6510 2023-11-30T21:48:21.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.554+0000 7fe1926c5700 1 -- 172.21.15.49:0/2311180904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe18c106e20 con 0x7fe18c0f6510 2023-11-30T21:48:21.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.554+0000 7fe188ff9700 1 -- 172.21.15.49:0/2311180904 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe17c007e70 con 0x7fe18c0f6510 2023-11-30T21:48:21.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.554+0000 7fe188ff9700 1 -- 172.21.15.49:0/2311180904 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe17c014ea0 con 0x7fe18c0f6510 2023-11-30T21:48:21.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.554+0000 7fe1926c5700 1 -- 172.21.15.49:0/2311180904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18c1072b0 con 0x7fe18c0f6510 2023-11-30T21:48:21.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.555+0000 7fe188ff9700 1 -- 172.21.15.49:0/2311180904 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe17c025070 con 0x7fe18c0f6510 2023-11-30T21:48:21.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.555+0000 7fe1926c5700 1 -- 172.21.15.49:0/2311180904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18c045ad0 con 0x7fe18c0f6510 2023-11-30T21:48:21.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.555+0000 7fe188ff9700 1 --2- 172.21.15.49:0/2311180904 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe17402ff60 0x7fe174032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:21.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.555+0000 7fe188ff9700 1 -- 172.21.15.49:0/2311180904 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe17c040920 con 0x7fe18c0f6510 2023-11-30T21:48:21.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.555+0000 7fe18b7fe700 1 --2- 172.21.15.49:0/2311180904 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe17402ff60 0x7fe174032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:21.558 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.556+0000 7fe18b7fe700 1 --2- 172.21.15.49:0/2311180904 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe17402ff60 0x7fe174032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fe1800096a0 tx=0x7fe180006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:21.562 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.560+0000 7fe188ff9700 1 -- 172.21.15.49:0/2311180904 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe17c00f4e0 con 0x7fe18c0f6510 2023-11-30T21:48:21.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.777+0000 7fe1926c5700 1 -- 172.21.15.49:0/2311180904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe18c0fbf60 con 0x7fe18c0f6510 2023-11-30T21:48:21.779 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.778+0000 7fe188ff9700 1 -- 172.21.15.49:0/2311180904 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe17c00f300 con 0x7fe18c0f6510 2023-11-30T21:48:21.779 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:21.779 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.779+0000 7fe1926c5700 1 -- 172.21.15.49:0/2311180904 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe17402ff60 msgr2=0x7fe174032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.779+0000 7fe1926c5700 1 --2- 172.21.15.49:0/2311180904 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe17402ff60 0x7fe174032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fe1800096a0 tx=0x7fe180006b40).stop 2023-11-30T21:48:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.779+0000 7fe1926c5700 1 -- 172.21.15.49:0/2311180904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe18c0f6510 msgr2=0x7fe18c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.779+0000 7fe1926c5700 1 --2- 172.21.15.49:0/2311180904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe18c0f6510 0x7fe18c106730 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fe17c014000 tx=0x7fe17c00b040).stop 2023-11-30T21:48:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.779+0000 7fe1926c5700 1 -- 172.21.15.49:0/2311180904 shutdown_connections 2023-11-30T21:48:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.779+0000 7fe1926c5700 1 --2- 172.21.15.49:0/2311180904 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe17402ff60 0x7fe174032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.779+0000 7fe1926c5700 1 --2- 172.21.15.49:0/2311180904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe18c0f6510 0x7fe18c106730 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.779+0000 7fe1926c5700 1 -- 172.21.15.49:0/2311180904 >> 172.21.15.49:0/2311180904 conn(0x7fe18c0f1c70 msgr2=0x7fe18c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.779+0000 7fe1926c5700 1 -- 172.21.15.49:0/2311180904 shutdown_connections 2023-11-30T21:48:21.781 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:21.779+0000 7fe1926c5700 1 -- 172.21.15.49:0/2311180904 wait complete. 2023-11-30T21:48:21.782 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:22.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:22 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:21.779139+0000 mon.smithi027 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.49:0/2311180904' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:23.170 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:23.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:23.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:23 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:21.498570+0000 mgr.smithi027.brrjcl (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:24.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.384+0000 7ff899104700 1 -- 172.21.15.49:0/4012671570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8940f62a0 msgr2=0x7ff8940f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:24.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.384+0000 7ff899104700 1 --2- 172.21.15.49:0/4012671570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8940f62a0 0x7ff8940f66c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7ff884009fb0 tx=0x7ff884009b30).stop 2023-11-30T21:48:24.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.384+0000 7ff899104700 1 -- 172.21.15.49:0/4012671570 shutdown_connections 2023-11-30T21:48:24.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.384+0000 7ff899104700 1 --2- 172.21.15.49:0/4012671570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8940f62a0 0x7ff8940f66c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:24.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.384+0000 7ff899104700 1 -- 172.21.15.49:0/4012671570 >> 172.21.15.49:0/4012671570 conn(0x7ff8940f1aa0 msgr2=0x7ff8940f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:24.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.384+0000 7ff899104700 1 -- 172.21.15.49:0/4012671570 shutdown_connections 2023-11-30T21:48:24.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.384+0000 7ff899104700 1 -- 172.21.15.49:0/4012671570 wait complete. 2023-11-30T21:48:24.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.385+0000 7ff899104700 1 Processor -- start 2023-11-30T21:48:24.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.385+0000 7ff899104700 1 -- start start 2023-11-30T21:48:24.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.385+0000 7ff899104700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8940f62a0 0x7ff8941064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:24.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.385+0000 7ff899104700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8941069a0 con 0x7ff8940f62a0 2023-11-30T21:48:24.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.385+0000 7ff893fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8940f62a0 0x7ff8941064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:24.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.385+0000 7ff893fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8940f62a0 0x7ff8941064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:34274/0 (socket says 172.21.15.49:34274) 2023-11-30T21:48:24.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.385+0000 7ff893fff700 1 -- 172.21.15.49:0/2531067164 learned_addr learned my addr 172.21.15.49:0/2531067164 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:24.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.386+0000 7ff893fff700 1 -- 172.21.15.49:0/2531067164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff884005040 con 0x7ff8940f62a0 2023-11-30T21:48:24.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.386+0000 7ff893fff700 1 --2- 172.21.15.49:0/2531067164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8940f62a0 0x7ff8941064b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7ff884013510 tx=0x7ff88400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:24.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.386+0000 7ff8917fa700 1 -- 172.21.15.49:0/2531067164 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff884009320 con 0x7ff8940f62a0 2023-11-30T21:48:24.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.386+0000 7ff899104700 1 -- 172.21.15.49:0/2531067164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff894106ba0 con 0x7ff8940f62a0 2023-11-30T21:48:24.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.386+0000 7ff8917fa700 1 -- 172.21.15.49:0/2531067164 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff884013f00 con 0x7ff8940f62a0 2023-11-30T21:48:24.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.386+0000 7ff8917fa700 1 -- 172.21.15.49:0/2531067164 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff884014dc0 con 0x7ff8940f62a0 2023-11-30T21:48:24.390 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.386+0000 7ff899104700 1 -- 172.21.15.49:0/2531067164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff894107030 con 0x7ff8940f62a0 2023-11-30T21:48:24.390 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.387+0000 7ff8917fa700 1 -- 172.21.15.49:0/2531067164 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff88401d3d0 con 0x7ff8940f62a0 2023-11-30T21:48:24.390 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.387+0000 7ff8917fa700 1 --2- 172.21.15.49:0/2531067164 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff88002ff60 0x7ff880032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:24.391 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.387+0000 7ff8917fa700 1 -- 172.21.15.49:0/2531067164 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8840410f0 con 0x7ff8940f62a0 2023-11-30T21:48:24.391 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.387+0000 7ff899104700 1 -- 172.21.15.49:0/2531067164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff894059100 con 0x7ff8940f62a0 2023-11-30T21:48:24.391 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.387+0000 7ff88bfff700 1 --2- 172.21.15.49:0/2531067164 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff88002ff60 0x7ff880032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:24.391 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.387+0000 7ff88bfff700 1 --2- 172.21.15.49:0/2531067164 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff88002ff60 0x7ff880032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7ff87c0096a0 tx=0x7ff87c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:24.393 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.392+0000 7ff8917fa700 1 -- 172.21.15.49:0/2531067164 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff884012920 con 0x7ff8940f62a0 2023-11-30T21:48:24.612 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.611+0000 7ff899104700 1 -- 172.21.15.49:0/2531067164 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8940fbce0 con 0x7ff8940f62a0 2023-11-30T21:48:24.613 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.611+0000 7ff8917fa700 1 -- 172.21.15.49:0/2531067164 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff88402b020 con 0x7ff8940f62a0 2023-11-30T21:48:24.613 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:24.613 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:24.615 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.613+0000 7ff899104700 1 -- 172.21.15.49:0/2531067164 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff88002ff60 msgr2=0x7ff880032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:24.615 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.613+0000 7ff899104700 1 --2- 172.21.15.49:0/2531067164 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff88002ff60 0x7ff880032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7ff87c0096a0 tx=0x7ff87c006b40).stop 2023-11-30T21:48:24.615 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.613+0000 7ff899104700 1 -- 172.21.15.49:0/2531067164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8940f62a0 msgr2=0x7ff8941064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:24.615 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.613+0000 7ff899104700 1 --2- 172.21.15.49:0/2531067164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8940f62a0 0x7ff8941064b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7ff884013510 tx=0x7ff88400b040).stop 2023-11-30T21:48:24.615 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.613+0000 7ff899104700 1 -- 172.21.15.49:0/2531067164 shutdown_connections 2023-11-30T21:48:24.615 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.613+0000 7ff899104700 1 --2- 172.21.15.49:0/2531067164 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff88002ff60 0x7ff880032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:24.615 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.613+0000 7ff899104700 1 --2- 172.21.15.49:0/2531067164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8940f62a0 0x7ff8941064b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:24.615 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.613+0000 7ff899104700 1 -- 172.21.15.49:0/2531067164 >> 172.21.15.49:0/2531067164 conn(0x7ff8940f1aa0 msgr2=0x7ff8940f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:24.615 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.613+0000 7ff899104700 1 -- 172.21.15.49:0/2531067164 shutdown_connections 2023-11-30T21:48:24.615 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:24.613+0000 7ff899104700 1 -- 172.21.15.49:0/2531067164 wait complete. 2023-11-30T21:48:24.616 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:25.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:23.498845+0000 mgr.smithi027.brrjcl (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:25.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:24.612824+0000 mon.smithi027 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.49:0/2531067164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:25.996 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:25.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:27.218 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.215+0000 7f84b6c3c700 1 -- 172.21.15.49:0/331851193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b00f62a0 msgr2=0x7f84b00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:27.218 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.215+0000 7f84b6c3c700 1 --2- 172.21.15.49:0/331851193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b00f62a0 0x7f84b00f66c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f84ac009fb0 tx=0x7f84ac009b30).stop 2023-11-30T21:48:27.218 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.215+0000 7f84b6c3c700 1 -- 172.21.15.49:0/331851193 shutdown_connections 2023-11-30T21:48:27.218 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.215+0000 7f84b6c3c700 1 --2- 172.21.15.49:0/331851193 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b00f62a0 0x7f84b00f66c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.215+0000 7f84b6c3c700 1 -- 172.21.15.49:0/331851193 >> 172.21.15.49:0/331851193 conn(0x7f84b00f1ac0 msgr2=0x7f84b00f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.215+0000 7f84b6c3c700 1 -- 172.21.15.49:0/331851193 shutdown_connections 2023-11-30T21:48:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.215+0000 7f84b6c3c700 1 -- 172.21.15.49:0/331851193 wait complete. 2023-11-30T21:48:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.215+0000 7f84b6c3c700 1 Processor -- start 2023-11-30T21:48:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.215+0000 7f84b6c3c700 1 -- start start 2023-11-30T21:48:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.216+0000 7f84b6c3c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b00f62a0 0x7f84b006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.216+0000 7f84b6c3c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b0069e20 con 0x7f84b00f62a0 2023-11-30T21:48:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.216+0000 7f84b5c3a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b00f62a0 0x7f84b006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.216+0000 7f84b5c3a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b00f62a0 0x7f84b006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:34280/0 (socket says 172.21.15.49:34280) 2023-11-30T21:48:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.216+0000 7f84b5c3a700 1 -- 172.21.15.49:0/3880767646 learned_addr learned my addr 172.21.15.49:0/3880767646 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.216+0000 7f84b5c3a700 1 -- 172.21.15.49:0/3880767646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ac005040 con 0x7f84b00f62a0 2023-11-30T21:48:27.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.216+0000 7f84b5c3a700 1 --2- 172.21.15.49:0/3880767646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b00f62a0 0x7f84b006b9a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f84ac013510 tx=0x7f84ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:27.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.216+0000 7f84a27fc700 1 -- 172.21.15.49:0/3880767646 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84ac009320 con 0x7f84b00f62a0 2023-11-30T21:48:27.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.216+0000 7f84a27fc700 1 -- 172.21.15.49:0/3880767646 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84ac013f00 con 0x7f84b00f62a0 2023-11-30T21:48:27.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.216+0000 7f84a27fc700 1 -- 172.21.15.49:0/3880767646 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84ac014dc0 con 0x7f84b00f62a0 2023-11-30T21:48:27.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.216+0000 7f84b6c3c700 1 -- 172.21.15.49:0/3880767646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b0069fc0 con 0x7f84b00f62a0 2023-11-30T21:48:27.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.217+0000 7f84b6c3c700 1 -- 172.21.15.49:0/3880767646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b006a450 con 0x7f84b00f62a0 2023-11-30T21:48:27.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.217+0000 7f84a27fc700 1 -- 172.21.15.49:0/3880767646 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84ac0095f0 con 0x7f84b00f62a0 2023-11-30T21:48:27.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.218+0000 7f84b6c3c700 1 -- 172.21.15.49:0/3880767646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b0059100 con 0x7f84b00f62a0 2023-11-30T21:48:27.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.218+0000 7f84a27fc700 1 --2- 172.21.15.49:0/3880767646 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f849c02ff60 0x7f849c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:27.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.218+0000 7f84a27fc700 1 -- 172.21.15.49:0/3880767646 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84ac041000 con 0x7f84b00f62a0 2023-11-30T21:48:27.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.218+0000 7f84b5439700 1 --2- 172.21.15.49:0/3880767646 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f849c02ff60 0x7f849c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:27.221 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.218+0000 7f84b5439700 1 --2- 172.21.15.49:0/3880767646 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f849c02ff60 0x7f849c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f84a40096a0 tx=0x7f84a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:27.224 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.222+0000 7f84a27fc700 1 -- 172.21.15.49:0/3880767646 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84ac014420 con 0x7f84b00f62a0 2023-11-30T21:48:27.439 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.438+0000 7f84b6c3c700 1 -- 172.21.15.49:0/3880767646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84b00fbce0 con 0x7f84b00f62a0 2023-11-30T21:48:27.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.439+0000 7f84a27fc700 1 -- 172.21.15.49:0/3880767646 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f84ac010230 con 0x7f84b00f62a0 2023-11-30T21:48:27.441 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:27.441 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:27.442 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.440+0000 7f84b6c3c700 1 -- 172.21.15.49:0/3880767646 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f849c02ff60 msgr2=0x7f849c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:27.442 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.440+0000 7f84b6c3c700 1 --2- 172.21.15.49:0/3880767646 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f849c02ff60 0x7f849c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f84a40096a0 tx=0x7f84a4006b40).stop 2023-11-30T21:48:27.443 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.440+0000 7f84b6c3c700 1 -- 172.21.15.49:0/3880767646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b00f62a0 msgr2=0x7f84b006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:27.443 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.440+0000 7f84b6c3c700 1 --2- 172.21.15.49:0/3880767646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b00f62a0 0x7f84b006b9a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f84ac013510 tx=0x7f84ac00b040).stop 2023-11-30T21:48:27.443 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.440+0000 7f84b6c3c700 1 -- 172.21.15.49:0/3880767646 shutdown_connections 2023-11-30T21:48:27.443 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.440+0000 7f84b6c3c700 1 --2- 172.21.15.49:0/3880767646 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f849c02ff60 0x7f849c032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:27.443 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.440+0000 7f84b6c3c700 1 --2- 172.21.15.49:0/3880767646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84b00f62a0 0x7f84b006b9a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:27.444 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.440+0000 7f84b6c3c700 1 -- 172.21.15.49:0/3880767646 >> 172.21.15.49:0/3880767646 conn(0x7f84b00f1ac0 msgr2=0x7f84b00f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:27.444 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.440+0000 7f84b6c3c700 1 -- 172.21.15.49:0/3880767646 shutdown_connections 2023-11-30T21:48:27.444 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:27.441+0000 7f84b6c3c700 1 -- 172.21.15.49:0/3880767646 wait complete. 2023-11-30T21:48:27.444 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:27.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:27 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:25.499165+0000 mgr.smithi027.brrjcl (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:28.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:28 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:27.440181+0000 mon.smithi027 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.49:0/3880767646' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:28.871 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:28.872 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:29.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:27.499417+0000 mgr.smithi027.brrjcl (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:30.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.100+0000 7f2ef4528700 1 -- 172.21.15.49:0/684876813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec060280 msgr2=0x7f2eec0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:30.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.100+0000 7f2ef4528700 1 --2- 172.21.15.49:0/684876813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec060280 0x7f2eec0f90f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f2ee8009fb0 tx=0x7f2ee8009b30).stop 2023-11-30T21:48:30.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.100+0000 7f2ef4528700 1 -- 172.21.15.49:0/684876813 shutdown_connections 2023-11-30T21:48:30.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.100+0000 7f2ef4528700 1 --2- 172.21.15.49:0/684876813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec060280 0x7f2eec0f90f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:30.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.100+0000 7f2ef4528700 1 -- 172.21.15.49:0/684876813 >> 172.21.15.49:0/684876813 conn(0x7f2eec0f0b70 msgr2=0x7f2eec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:30.102 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.100+0000 7f2ef4528700 1 -- 172.21.15.49:0/684876813 shutdown_connections 2023-11-30T21:48:30.103 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.100+0000 7f2ef4528700 1 -- 172.21.15.49:0/684876813 wait complete. 2023-11-30T21:48:30.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.101+0000 7f2ef4528700 1 Processor -- start 2023-11-30T21:48:30.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.101+0000 7f2ef4528700 1 -- start start 2023-11-30T21:48:30.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.101+0000 7f2ef4528700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec060280 0x7f2eec1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:30.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.101+0000 7f2ef4528700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec106bd0 con 0x7f2eec060280 2023-11-30T21:48:30.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.101+0000 7f2ef22c4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec060280 0x7f2eec1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:30.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.101+0000 7f2ef22c4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec060280 0x7f2eec1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59776/0 (socket says 172.21.15.49:59776) 2023-11-30T21:48:30.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.101+0000 7f2ef22c4700 1 -- 172.21.15.49:0/3091435591 learned_addr learned my addr 172.21.15.49:0/3091435591 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:30.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.102+0000 7f2ef22c4700 1 -- 172.21.15.49:0/3091435591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee8005040 con 0x7f2eec060280 2023-11-30T21:48:30.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.102+0000 7f2ef22c4700 1 --2- 172.21.15.49:0/3091435591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec060280 0x7f2eec1066e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f2ee8013510 tx=0x7f2ee800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:30.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.102+0000 7f2edeffd700 1 -- 172.21.15.49:0/3091435591 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ee8009320 con 0x7f2eec060280 2023-11-30T21:48:30.105 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.102+0000 7f2ef4528700 1 -- 172.21.15.49:0/3091435591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eec106dd0 con 0x7f2eec060280 2023-11-30T21:48:30.105 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.102+0000 7f2edeffd700 1 -- 172.21.15.49:0/3091435591 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ee8013f00 con 0x7f2eec060280 2023-11-30T21:48:30.105 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.102+0000 7f2edeffd700 1 -- 172.21.15.49:0/3091435591 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ee8014dc0 con 0x7f2eec060280 2023-11-30T21:48:30.105 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.102+0000 7f2ef4528700 1 -- 172.21.15.49:0/3091435591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eec107260 con 0x7f2eec060280 2023-11-30T21:48:30.106 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.103+0000 7f2ef4528700 1 -- 172.21.15.49:0/3091435591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eec045ad0 con 0x7f2eec060280 2023-11-30T21:48:30.106 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.103+0000 7f2edeffd700 1 -- 172.21.15.49:0/3091435591 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ee80095f0 con 0x7f2eec060280 2023-11-30T21:48:30.106 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.103+0000 7f2edeffd700 1 --2- 172.21.15.49:0/3091435591 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2ed802ff60 0x7f2ed8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:30.106 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.103+0000 7f2edeffd700 1 -- 172.21.15.49:0/3091435591 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ee8041030 con 0x7f2eec060280 2023-11-30T21:48:30.106 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.103+0000 7f2ef1ac3700 1 --2- 172.21.15.49:0/3091435591 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2ed802ff60 0x7f2ed8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:30.106 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.104+0000 7f2ef1ac3700 1 --2- 172.21.15.49:0/3091435591 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2ed802ff60 0x7f2ed8032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f2ee00096a0 tx=0x7f2ee0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:30.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.108+0000 7f2edeffd700 1 -- 172.21.15.49:0/3091435591 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ee8014420 con 0x7f2eec060280 2023-11-30T21:48:30.328 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.327+0000 7f2ef4528700 1 -- 172.21.15.49:0/3091435591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2eec108110 con 0x7f2eec060280 2023-11-30T21:48:30.329 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.327+0000 7f2edeffd700 1 -- 172.21.15.49:0/3091435591 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2ee8010230 con 0x7f2eec060280 2023-11-30T21:48:30.329 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:30.329 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:30.330 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.329+0000 7f2ef4528700 1 -- 172.21.15.49:0/3091435591 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2ed802ff60 msgr2=0x7f2ed8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:30.330 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.329+0000 7f2ef4528700 1 --2- 172.21.15.49:0/3091435591 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2ed802ff60 0x7f2ed8032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f2ee00096a0 tx=0x7f2ee0006b40).stop 2023-11-30T21:48:30.331 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.329+0000 7f2ef4528700 1 -- 172.21.15.49:0/3091435591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec060280 msgr2=0x7f2eec1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:30.331 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.329+0000 7f2ef4528700 1 --2- 172.21.15.49:0/3091435591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec060280 0x7f2eec1066e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f2ee8013510 tx=0x7f2ee800b040).stop 2023-11-30T21:48:30.331 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.329+0000 7f2ef4528700 1 -- 172.21.15.49:0/3091435591 shutdown_connections 2023-11-30T21:48:30.331 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.329+0000 7f2ef4528700 1 --2- 172.21.15.49:0/3091435591 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f2ed802ff60 0x7f2ed8032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:30.331 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.329+0000 7f2ef4528700 1 --2- 172.21.15.49:0/3091435591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec060280 0x7f2eec1066e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:30.331 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.329+0000 7f2ef4528700 1 -- 172.21.15.49:0/3091435591 >> 172.21.15.49:0/3091435591 conn(0x7f2eec0f0b70 msgr2=0x7f2eec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:30.331 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.329+0000 7f2ef4528700 1 -- 172.21.15.49:0/3091435591 shutdown_connections 2023-11-30T21:48:30.331 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:30.329+0000 7f2ef4528700 1 -- 172.21.15.49:0/3091435591 wait complete. 2023-11-30T21:48:30.331 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:31.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:29.499760+0000 mgr.smithi027.brrjcl (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:31.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:30.328657+0000 mon.smithi027 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.49:0/3091435591' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:31.767 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:31.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:33.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.033+0000 7fee3713a700 1 -- 172.21.15.49:0/296935690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee300601e0 msgr2=0x7fee300f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:33.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.033+0000 7fee3713a700 1 --2- 172.21.15.49:0/296935690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee300601e0 0x7fee300f8f20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fee2c009fb0 tx=0x7fee2c009b30).stop 2023-11-30T21:48:33.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.033+0000 7fee3713a700 1 -- 172.21.15.49:0/296935690 shutdown_connections 2023-11-30T21:48:33.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.033+0000 7fee3713a700 1 --2- 172.21.15.49:0/296935690 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee300601e0 0x7fee300f8f20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:33.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.033+0000 7fee3713a700 1 -- 172.21.15.49:0/296935690 >> 172.21.15.49:0/296935690 conn(0x7fee300f09a0 msgr2=0x7fee300f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:33.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.033+0000 7fee3713a700 1 -- 172.21.15.49:0/296935690 shutdown_connections 2023-11-30T21:48:33.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.033+0000 7fee3713a700 1 -- 172.21.15.49:0/296935690 wait complete. 2023-11-30T21:48:33.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.033+0000 7fee3713a700 1 Processor -- start 2023-11-30T21:48:33.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.034+0000 7fee3713a700 1 -- start start 2023-11-30T21:48:33.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.034+0000 7fee3713a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee300601e0 0x7fee30106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:33.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.034+0000 7fee3713a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee301069f0 con 0x7fee300601e0 2023-11-30T21:48:33.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.034+0000 7fee36138700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee300601e0 0x7fee30106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:33.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.034+0000 7fee36138700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee300601e0 0x7fee30106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59796/0 (socket says 172.21.15.49:59796) 2023-11-30T21:48:33.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.034+0000 7fee36138700 1 -- 172.21.15.49:0/1961096259 learned_addr learned my addr 172.21.15.49:0/1961096259 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:33.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.034+0000 7fee36138700 1 -- 172.21.15.49:0/1961096259 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c005040 con 0x7fee300601e0 2023-11-30T21:48:33.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.034+0000 7fee36138700 1 --2- 172.21.15.49:0/1961096259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee300601e0 0x7fee30106500 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fee2c013510 tx=0x7fee2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:33.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.035+0000 7fee26ffd700 1 -- 172.21.15.49:0/1961096259 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee2c009320 con 0x7fee300601e0 2023-11-30T21:48:33.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.035+0000 7fee3713a700 1 -- 172.21.15.49:0/1961096259 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee30106bf0 con 0x7fee300601e0 2023-11-30T21:48:33.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.035+0000 7fee26ffd700 1 -- 172.21.15.49:0/1961096259 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee2c013f00 con 0x7fee300601e0 2023-11-30T21:48:33.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.035+0000 7fee26ffd700 1 -- 172.21.15.49:0/1961096259 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee2c014d90 con 0x7fee300601e0 2023-11-30T21:48:33.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.035+0000 7fee3713a700 1 -- 172.21.15.49:0/1961096259 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee30107080 con 0x7fee300601e0 2023-11-30T21:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.035+0000 7fee26ffd700 1 -- 172.21.15.49:0/1961096259 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee2c0095f0 con 0x7fee300601e0 2023-11-30T21:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.035+0000 7fee3713a700 1 -- 172.21.15.49:0/1961096259 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee30045ad0 con 0x7fee300601e0 2023-11-30T21:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.036+0000 7fee26ffd700 1 --2- 172.21.15.49:0/1961096259 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fee1c02ff60 0x7fee1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.036+0000 7fee26ffd700 1 -- 172.21.15.49:0/1961096259 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee2c040fe0 con 0x7fee300601e0 2023-11-30T21:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.036+0000 7fee35937700 1 --2- 172.21.15.49:0/1961096259 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fee1c02ff60 0x7fee1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.036+0000 7fee35937700 1 --2- 172.21.15.49:0/1961096259 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fee1c02ff60 0x7fee1c032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fee200096a0 tx=0x7fee20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:33.042 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.040+0000 7fee26ffd700 1 -- 172.21.15.49:0/1961096259 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee2c01d7f0 con 0x7fee300601e0 2023-11-30T21:48:33.259 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.258+0000 7fee3713a700 1 -- 172.21.15.49:0/1961096259 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee30107f30 con 0x7fee300601e0 2023-11-30T21:48:33.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.259+0000 7fee26ffd700 1 -- 172.21.15.49:0/1961096259 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee2c010230 con 0x7fee300601e0 2023-11-30T21:48:33.261 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:33.261 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:33.262 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.260+0000 7fee3713a700 1 -- 172.21.15.49:0/1961096259 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fee1c02ff60 msgr2=0x7fee1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:33.262 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.260+0000 7fee3713a700 1 --2- 172.21.15.49:0/1961096259 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fee1c02ff60 0x7fee1c032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fee200096a0 tx=0x7fee20006b40).stop 2023-11-30T21:48:33.262 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.260+0000 7fee3713a700 1 -- 172.21.15.49:0/1961096259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee300601e0 msgr2=0x7fee30106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:33.262 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.260+0000 7fee3713a700 1 --2- 172.21.15.49:0/1961096259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee300601e0 0x7fee30106500 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fee2c013510 tx=0x7fee2c00b040).stop 2023-11-30T21:48:33.262 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.260+0000 7fee3713a700 1 -- 172.21.15.49:0/1961096259 shutdown_connections 2023-11-30T21:48:33.262 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.260+0000 7fee3713a700 1 --2- 172.21.15.49:0/1961096259 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fee1c02ff60 0x7fee1c032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:33.263 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.260+0000 7fee3713a700 1 --2- 172.21.15.49:0/1961096259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee300601e0 0x7fee30106500 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:33.263 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.260+0000 7fee3713a700 1 -- 172.21.15.49:0/1961096259 >> 172.21.15.49:0/1961096259 conn(0x7fee300f09a0 msgr2=0x7fee300f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:33.263 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.260+0000 7fee3713a700 1 -- 172.21.15.49:0/1961096259 shutdown_connections 2023-11-30T21:48:33.263 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:33.260+0000 7fee3713a700 1 -- 172.21.15.49:0/1961096259 wait complete. 2023-11-30T21:48:33.263 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:33.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:33 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:31.500025+0000 mgr.smithi027.brrjcl (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:34.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:34 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:33.259990+0000 mon.smithi027 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.49:0/1961096259' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:34.626 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:34.626 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:35.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:35 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:33.500221+0000 mgr.smithi027.brrjcl (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:35.890 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.888+0000 7f12f5300700 1 -- 172.21.15.49:0/1354271977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f00f47c0 msgr2=0x7f12f00f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:35.890 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.888+0000 7f12f5300700 1 --2- 172.21.15.49:0/1354271977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f00f47c0 0x7f12f00f4be0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f12e4009fb0 tx=0x7f12e4009b30).stop 2023-11-30T21:48:35.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.888+0000 7f12f5300700 1 -- 172.21.15.49:0/1354271977 shutdown_connections 2023-11-30T21:48:35.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.888+0000 7f12f5300700 1 --2- 172.21.15.49:0/1354271977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f00f47c0 0x7f12f00f4be0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:35.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.888+0000 7f12f5300700 1 -- 172.21.15.49:0/1354271977 >> 172.21.15.49:0/1354271977 conn(0x7f12f00f02e0 msgr2=0x7f12f00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:35.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.888+0000 7f12f5300700 1 -- 172.21.15.49:0/1354271977 shutdown_connections 2023-11-30T21:48:35.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.888+0000 7f12f5300700 1 -- 172.21.15.49:0/1354271977 wait complete. 2023-11-30T21:48:35.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.889+0000 7f12f5300700 1 Processor -- start 2023-11-30T21:48:35.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.889+0000 7f12f5300700 1 -- start start 2023-11-30T21:48:35.892 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.889+0000 7f12f5300700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f00f47c0 0x7f12f00f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:35.892 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.889+0000 7f12f5300700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f00f57f0 con 0x7f12f00f47c0 2023-11-30T21:48:35.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.889+0000 7f12eeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f00f47c0 0x7f12f00f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:35.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.889+0000 7f12eeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f00f47c0 0x7f12f00f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59818/0 (socket says 172.21.15.49:59818) 2023-11-30T21:48:35.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.889+0000 7f12eeffd700 1 -- 172.21.15.49:0/2168054368 learned_addr learned my addr 172.21.15.49:0/2168054368 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:35.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.890+0000 7f12eeffd700 1 -- 172.21.15.49:0/2168054368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e4005040 con 0x7f12f00f47c0 2023-11-30T21:48:35.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.890+0000 7f12eeffd700 1 --2- 172.21.15.49:0/2168054368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f00f47c0 0x7f12f00f7370 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f12e4013510 tx=0x7f12e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:35.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.890+0000 7f12d77fe700 1 -- 172.21.15.49:0/2168054368 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12e4009320 con 0x7f12f00f47c0 2023-11-30T21:48:35.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.890+0000 7f12d77fe700 1 -- 172.21.15.49:0/2168054368 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12e4013f00 con 0x7f12f00f47c0 2023-11-30T21:48:35.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.890+0000 7f12f5300700 1 -- 172.21.15.49:0/2168054368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f00f5990 con 0x7f12f00f47c0 2023-11-30T21:48:35.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.890+0000 7f12d77fe700 1 -- 172.21.15.49:0/2168054368 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12e4014d90 con 0x7f12f00f47c0 2023-11-30T21:48:35.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.890+0000 7f12f5300700 1 -- 172.21.15.49:0/2168054368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f00f5e20 con 0x7f12f00f47c0 2023-11-30T21:48:35.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.891+0000 7f12d77fe700 1 -- 172.21.15.49:0/2168054368 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12e4014420 con 0x7f12f00f47c0 2023-11-30T21:48:35.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.891+0000 7f12f5300700 1 -- 172.21.15.49:0/2168054368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f0059100 con 0x7f12f00f47c0 2023-11-30T21:48:35.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.891+0000 7f12d77fe700 1 --2- 172.21.15.49:0/2168054368 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f12d802ff60 0x7f12d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:35.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.891+0000 7f12d77fe700 1 -- 172.21.15.49:0/2168054368 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12e4041000 con 0x7f12f00f47c0 2023-11-30T21:48:35.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.891+0000 7f12ee7fc700 1 --2- 172.21.15.49:0/2168054368 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f12d802ff60 0x7f12d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:35.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.891+0000 7f12ee7fc700 1 --2- 172.21.15.49:0/2168054368 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f12d802ff60 0x7f12d8032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f12e00096a0 tx=0x7f12e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:35.897 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:35.896+0000 7f12d77fe700 1 -- 172.21.15.49:0/2168054368 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12e40095f0 con 0x7f12f00f47c0 2023-11-30T21:48:36.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:36.115+0000 7f12f5300700 1 -- 172.21.15.49:0/2168054368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12f00fbf60 con 0x7f12f00f47c0 2023-11-30T21:48:36.117 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:36.116+0000 7f12d77fe700 1 -- 172.21.15.49:0/2168054368 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12e401db70 con 0x7f12f00f47c0 2023-11-30T21:48:36.117 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:36.117 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:36.119 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:36.117+0000 7f12f5300700 1 -- 172.21.15.49:0/2168054368 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f12d802ff60 msgr2=0x7f12d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:36.119 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:36.117+0000 7f12f5300700 1 --2- 172.21.15.49:0/2168054368 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f12d802ff60 0x7f12d8032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f12e00096a0 tx=0x7f12e0006b40).stop 2023-11-30T21:48:36.119 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:36.117+0000 7f12f5300700 1 -- 172.21.15.49:0/2168054368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f00f47c0 msgr2=0x7f12f00f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:36.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:36.117+0000 7f12f5300700 1 --2- 172.21.15.49:0/2168054368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f00f47c0 0x7f12f00f7370 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f12e4013510 tx=0x7f12e400b040).stop 2023-11-30T21:48:36.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:36.117+0000 7f12f5300700 1 -- 172.21.15.49:0/2168054368 shutdown_connections 2023-11-30T21:48:36.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:36.117+0000 7f12f5300700 1 --2- 172.21.15.49:0/2168054368 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f12d802ff60 0x7f12d8032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:36.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:36.117+0000 7f12f5300700 1 --2- 172.21.15.49:0/2168054368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12f00f47c0 0x7f12f00f7370 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:36.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:36.117+0000 7f12f5300700 1 -- 172.21.15.49:0/2168054368 >> 172.21.15.49:0/2168054368 conn(0x7f12f00f02e0 msgr2=0x7f12f00f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:36.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:36.117+0000 7f12f5300700 1 -- 172.21.15.49:0/2168054368 shutdown_connections 2023-11-30T21:48:36.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:36.117+0000 7f12f5300700 1 -- 172.21.15.49:0/2168054368 wait complete. 2023-11-30T21:48:36.120 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:36.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:36.117186+0000 mon.smithi027 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.49:0/2168054368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:37.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:37 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:35.500553+0000 mgr.smithi027.brrjcl (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:37.516 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:37.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:38.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.792+0000 7f38be4f9700 1 -- 172.21.15.49:0/2638343442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b80f5410 msgr2=0x7f38b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:38.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.792+0000 7f38be4f9700 1 --2- 172.21.15.49:0/2638343442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b80f5410 0x7f38b80f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f38a8009fb0 tx=0x7f38a8009b30).stop 2023-11-30T21:48:38.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.792+0000 7f38be4f9700 1 -- 172.21.15.49:0/2638343442 shutdown_connections 2023-11-30T21:48:38.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.792+0000 7f38be4f9700 1 --2- 172.21.15.49:0/2638343442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b80f5410 0x7f38b80f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:38.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.792+0000 7f38be4f9700 1 -- 172.21.15.49:0/2638343442 >> 172.21.15.49:0/2638343442 conn(0x7f38b80f0b70 msgr2=0x7f38b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:38.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.792+0000 7f38be4f9700 1 -- 172.21.15.49:0/2638343442 shutdown_connections 2023-11-30T21:48:38.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.792+0000 7f38be4f9700 1 -- 172.21.15.49:0/2638343442 wait complete. 2023-11-30T21:48:38.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.793+0000 7f38be4f9700 1 Processor -- start 2023-11-30T21:48:38.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.793+0000 7f38be4f9700 1 -- start start 2023-11-30T21:48:38.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.793+0000 7f38be4f9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b80f5410 0x7f38b8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:38.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.793+0000 7f38be4f9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b8105b20 con 0x7f38b80f5410 2023-11-30T21:48:38.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.793+0000 7f38b7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b80f5410 0x7f38b8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:38.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.794+0000 7f38b7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b80f5410 0x7f38b8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:57190/0 (socket says 172.21.15.49:57190) 2023-11-30T21:48:38.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.794+0000 7f38b7fff700 1 -- 172.21.15.49:0/42801564 learned_addr learned my addr 172.21.15.49:0/42801564 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:38.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.794+0000 7f38b7fff700 1 -- 172.21.15.49:0/42801564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a8005040 con 0x7f38b80f5410 2023-11-30T21:48:38.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.794+0000 7f38b7fff700 1 --2- 172.21.15.49:0/42801564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b80f5410 0x7f38b8105630 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f38a8013510 tx=0x7f38a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:38.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.794+0000 7f38b4ff9700 1 -- 172.21.15.49:0/42801564 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38a8009320 con 0x7f38b80f5410 2023-11-30T21:48:38.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.794+0000 7f38be4f9700 1 -- 172.21.15.49:0/42801564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b8105d20 con 0x7f38b80f5410 2023-11-30T21:48:38.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.794+0000 7f38b4ff9700 1 -- 172.21.15.49:0/42801564 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38a8013f00 con 0x7f38b80f5410 2023-11-30T21:48:38.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.794+0000 7f38b4ff9700 1 -- 172.21.15.49:0/42801564 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38a8014d90 con 0x7f38b80f5410 2023-11-30T21:48:38.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.794+0000 7f38be4f9700 1 -- 172.21.15.49:0/42801564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b81061b0 con 0x7f38b80f5410 2023-11-30T21:48:38.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.795+0000 7f38b4ff9700 1 -- 172.21.15.49:0/42801564 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38a8014420 con 0x7f38b80f5410 2023-11-30T21:48:38.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.795+0000 7f38be4f9700 1 -- 172.21.15.49:0/42801564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b8045ad0 con 0x7f38b80f5410 2023-11-30T21:48:38.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.795+0000 7f38b4ff9700 1 --2- 172.21.15.49:0/42801564 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38a002ff60 0x7f38a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:38.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.795+0000 7f38b4ff9700 1 -- 172.21.15.49:0/42801564 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38a8041070 con 0x7f38b80f5410 2023-11-30T21:48:38.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.796+0000 7f38b77fe700 1 --2- 172.21.15.49:0/42801564 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38a002ff60 0x7f38a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:38.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.796+0000 7f38b77fe700 1 --2- 172.21.15.49:0/42801564 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38a002ff60 0x7f38a0032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f38ac0096a0 tx=0x7f38ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:38.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:38.800+0000 7f38b4ff9700 1 -- 172.21.15.49:0/42801564 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38a80095f0 con 0x7f38b80f5410 2023-11-30T21:48:39.021 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:39.020+0000 7f38be4f9700 1 -- 172.21.15.49:0/42801564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38b80fae60 con 0x7f38b80f5410 2023-11-30T21:48:39.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:39.020+0000 7f38b4ff9700 1 -- 172.21.15.49:0/42801564 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f38a801db70 con 0x7f38b80f5410 2023-11-30T21:48:39.022 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:39.023 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:39.023 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:39.022+0000 7f38be4f9700 1 -- 172.21.15.49:0/42801564 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38a002ff60 msgr2=0x7f38a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:39.023 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:39.022+0000 7f38be4f9700 1 --2- 172.21.15.49:0/42801564 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38a002ff60 0x7f38a0032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f38ac0096a0 tx=0x7f38ac006b40).stop 2023-11-30T21:48:39.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:39.022+0000 7f38be4f9700 1 -- 172.21.15.49:0/42801564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b80f5410 msgr2=0x7f38b8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:39.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:39.022+0000 7f38be4f9700 1 --2- 172.21.15.49:0/42801564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b80f5410 0x7f38b8105630 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f38a8013510 tx=0x7f38a800b040).stop 2023-11-30T21:48:39.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:39.022+0000 7f38be4f9700 1 -- 172.21.15.49:0/42801564 shutdown_connections 2023-11-30T21:48:39.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:39.022+0000 7f38be4f9700 1 --2- 172.21.15.49:0/42801564 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38a002ff60 0x7f38a0032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:39.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:39.022+0000 7f38be4f9700 1 --2- 172.21.15.49:0/42801564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b80f5410 0x7f38b8105630 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:39.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:39.022+0000 7f38be4f9700 1 -- 172.21.15.49:0/42801564 >> 172.21.15.49:0/42801564 conn(0x7f38b80f0b70 msgr2=0x7f38b80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:39.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:39.022+0000 7f38be4f9700 1 -- 172.21.15.49:0/42801564 shutdown_connections 2023-11-30T21:48:39.024 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:39.022+0000 7f38be4f9700 1 -- 172.21.15.49:0/42801564 wait complete. 2023-11-30T21:48:39.024 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:39.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:37.500897+0000 mgr.smithi027.brrjcl (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:39.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:39.021846+0000 mon.smithi027 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.49:0/42801564' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:40.420 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:40.420 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:41.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:41 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:39.501289+0000 mgr.smithi027.brrjcl (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:41.651 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.648+0000 7f9d76a16700 1 -- 172.21.15.49:0/1709644772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d700f5410 msgr2=0x7f9d700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:41.651 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.648+0000 7f9d76a16700 1 --2- 172.21.15.49:0/1709644772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d700f5410 0x7f9d700f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f9d60009fb0 tx=0x7f9d60009b30).stop 2023-11-30T21:48:41.651 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.648+0000 7f9d76a16700 1 -- 172.21.15.49:0/1709644772 shutdown_connections 2023-11-30T21:48:41.651 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.648+0000 7f9d76a16700 1 --2- 172.21.15.49:0/1709644772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d700f5410 0x7f9d700f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:41.651 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.648+0000 7f9d76a16700 1 -- 172.21.15.49:0/1709644772 >> 172.21.15.49:0/1709644772 conn(0x7f9d700f0b70 msgr2=0x7f9d700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:41.651 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.648+0000 7f9d76a16700 1 -- 172.21.15.49:0/1709644772 shutdown_connections 2023-11-30T21:48:41.651 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.648+0000 7f9d76a16700 1 -- 172.21.15.49:0/1709644772 wait complete. 2023-11-30T21:48:41.651 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.649+0000 7f9d76a16700 1 Processor -- start 2023-11-30T21:48:41.651 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.649+0000 7f9d76a16700 1 -- start start 2023-11-30T21:48:41.652 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.649+0000 7f9d76a16700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d700f5410 0x7f9d700fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:41.652 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.649+0000 7f9d76a16700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d700fb820 con 0x7f9d700f5410 2023-11-30T21:48:41.653 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.650+0000 7f9d6ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d700f5410 0x7f9d700fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:41.653 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.650+0000 7f9d6ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d700f5410 0x7f9d700fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:57206/0 (socket says 172.21.15.49:57206) 2023-11-30T21:48:41.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.650+0000 7f9d6ffff700 1 -- 172.21.15.49:0/1067034953 learned_addr learned my addr 172.21.15.49:0/1067034953 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:41.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.650+0000 7f9d6ffff700 1 -- 172.21.15.49:0/1067034953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d60005040 con 0x7f9d700f5410 2023-11-30T21:48:41.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.650+0000 7f9d6ffff700 1 --2- 172.21.15.49:0/1067034953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d700f5410 0x7f9d700fd3a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f9d60013510 tx=0x7f9d6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:41.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.650+0000 7f9d6cff9700 1 -- 172.21.15.49:0/1067034953 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d60009320 con 0x7f9d700f5410 2023-11-30T21:48:41.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.651+0000 7f9d6cff9700 1 -- 172.21.15.49:0/1067034953 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d60013f00 con 0x7f9d700f5410 2023-11-30T21:48:41.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.651+0000 7f9d76a16700 1 -- 172.21.15.49:0/1067034953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d700fb9c0 con 0x7f9d700f5410 2023-11-30T21:48:41.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.651+0000 7f9d6cff9700 1 -- 172.21.15.49:0/1067034953 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d60014d90 con 0x7f9d700f5410 2023-11-30T21:48:41.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.651+0000 7f9d76a16700 1 -- 172.21.15.49:0/1067034953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d700fbe50 con 0x7f9d700f5410 2023-11-30T21:48:41.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.651+0000 7f9d76a16700 1 -- 172.21.15.49:0/1067034953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d70059100 con 0x7f9d700f5410 2023-11-30T21:48:41.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.651+0000 7f9d6cff9700 1 -- 172.21.15.49:0/1067034953 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d600095f0 con 0x7f9d700f5410 2023-11-30T21:48:41.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.652+0000 7f9d6cff9700 1 --2- 172.21.15.49:0/1067034953 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d5802ffb0 0x7f9d58032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:41.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.652+0000 7f9d6cff9700 1 -- 172.21.15.49:0/1067034953 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d60040f80 con 0x7f9d700f5410 2023-11-30T21:48:41.655 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.652+0000 7f9d6f7fe700 1 --2- 172.21.15.49:0/1067034953 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d5802ffb0 0x7f9d58032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:41.655 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.652+0000 7f9d6f7fe700 1 --2- 172.21.15.49:0/1067034953 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d5802ffb0 0x7f9d58032470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f9d640096a0 tx=0x7f9d64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:41.658 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.657+0000 7f9d6cff9700 1 -- 172.21.15.49:0/1067034953 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d60014420 con 0x7f9d700f5410 2023-11-30T21:48:41.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.874+0000 7f9d76a16700 1 -- 172.21.15.49:0/1067034953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d700fae60 con 0x7f9d700f5410 2023-11-30T21:48:41.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.875+0000 7f9d6cff9700 1 -- 172.21.15.49:0/1067034953 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9d6001d7f0 con 0x7f9d700f5410 2023-11-30T21:48:41.877 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:41.877 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:41.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.878+0000 7f9d76a16700 1 -- 172.21.15.49:0/1067034953 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d5802ffb0 msgr2=0x7f9d58032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:41.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.878+0000 7f9d76a16700 1 --2- 172.21.15.49:0/1067034953 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d5802ffb0 0x7f9d58032470 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f9d640096a0 tx=0x7f9d64006b40).stop 2023-11-30T21:48:41.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.878+0000 7f9d76a16700 1 -- 172.21.15.49:0/1067034953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d700f5410 msgr2=0x7f9d700fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:41.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.878+0000 7f9d76a16700 1 --2- 172.21.15.49:0/1067034953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d700f5410 0x7f9d700fd3a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f9d60013510 tx=0x7f9d6000b040).stop 2023-11-30T21:48:41.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.878+0000 7f9d76a16700 1 -- 172.21.15.49:0/1067034953 shutdown_connections 2023-11-30T21:48:41.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.878+0000 7f9d76a16700 1 --2- 172.21.15.49:0/1067034953 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d5802ffb0 0x7f9d58032470 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:41.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.878+0000 7f9d76a16700 1 --2- 172.21.15.49:0/1067034953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d700f5410 0x7f9d700fd3a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:41.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.878+0000 7f9d76a16700 1 -- 172.21.15.49:0/1067034953 >> 172.21.15.49:0/1067034953 conn(0x7f9d700f0b70 msgr2=0x7f9d7005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:41.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.878+0000 7f9d76a16700 1 -- 172.21.15.49:0/1067034953 shutdown_connections 2023-11-30T21:48:41.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:41.878+0000 7f9d76a16700 1 -- 172.21.15.49:0/1067034953 wait complete. 2023-11-30T21:48:41.880 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:42.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:42 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:41.876692+0000 mon.smithi027 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.49:0/1067034953' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:43.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:43.273 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:43.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:41.501628+0000 mgr.smithi027.brrjcl (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:44.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.522+0000 7fe934b6a700 1 -- 172.21.15.49:0/4084070311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9300f6510 msgr2=0x7fe9300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:44.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.522+0000 7fe934b6a700 1 --2- 172.21.15.49:0/4084070311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9300f6510 0x7fe9300f6930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fe920009fb0 tx=0x7fe920009b30).stop 2023-11-30T21:48:44.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.522+0000 7fe934b6a700 1 -- 172.21.15.49:0/4084070311 shutdown_connections 2023-11-30T21:48:44.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.522+0000 7fe934b6a700 1 --2- 172.21.15.49:0/4084070311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9300f6510 0x7fe9300f6930 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:44.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.522+0000 7fe934b6a700 1 -- 172.21.15.49:0/4084070311 >> 172.21.15.49:0/4084070311 conn(0x7fe9300f1c70 msgr2=0x7fe9300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:44.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.522+0000 7fe934b6a700 1 -- 172.21.15.49:0/4084070311 shutdown_connections 2023-11-30T21:48:44.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.522+0000 7fe934b6a700 1 -- 172.21.15.49:0/4084070311 wait complete. 2023-11-30T21:48:44.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.523+0000 7fe934b6a700 1 Processor -- start 2023-11-30T21:48:44.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.523+0000 7fe934b6a700 1 -- start start 2023-11-30T21:48:44.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.523+0000 7fe934b6a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9300f6510 0x7fe9301066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:44.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.523+0000 7fe934b6a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe930106bc0 con 0x7fe9300f6510 2023-11-30T21:48:44.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.523+0000 7fe92e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9300f6510 0x7fe9301066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:44.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.523+0000 7fe92e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9300f6510 0x7fe9301066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:57222/0 (socket says 172.21.15.49:57222) 2023-11-30T21:48:44.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.523+0000 7fe92e59c700 1 -- 172.21.15.49:0/2551002850 learned_addr learned my addr 172.21.15.49:0/2551002850 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:44.526 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.523+0000 7fe92e59c700 1 -- 172.21.15.49:0/2551002850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe920005040 con 0x7fe9300f6510 2023-11-30T21:48:44.526 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.523+0000 7fe92e59c700 1 --2- 172.21.15.49:0/2551002850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9300f6510 0x7fe9301066d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fe920013510 tx=0x7fe92000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:44.526 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.524+0000 7fe91effd700 1 -- 172.21.15.49:0/2551002850 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe920009320 con 0x7fe9300f6510 2023-11-30T21:48:44.526 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.524+0000 7fe934b6a700 1 -- 172.21.15.49:0/2551002850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe930106dc0 con 0x7fe9300f6510 2023-11-30T21:48:44.526 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.524+0000 7fe91effd700 1 -- 172.21.15.49:0/2551002850 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe920013f00 con 0x7fe9300f6510 2023-11-30T21:48:44.526 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.524+0000 7fe91effd700 1 -- 172.21.15.49:0/2551002850 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe920014d90 con 0x7fe9300f6510 2023-11-30T21:48:44.526 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.524+0000 7fe934b6a700 1 -- 172.21.15.49:0/2551002850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe930107250 con 0x7fe9300f6510 2023-11-30T21:48:44.527 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.525+0000 7fe91effd700 1 -- 172.21.15.49:0/2551002850 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9200095f0 con 0x7fe9300f6510 2023-11-30T21:48:44.527 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.525+0000 7fe934b6a700 1 -- 172.21.15.49:0/2551002850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe930045ad0 con 0x7fe9300f6510 2023-11-30T21:48:44.527 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.525+0000 7fe91effd700 1 --2- 172.21.15.49:0/2551002850 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe91802ff60 0x7fe918032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:44.527 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.525+0000 7fe91effd700 1 -- 172.21.15.49:0/2551002850 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe920041000 con 0x7fe9300f6510 2023-11-30T21:48:44.527 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.525+0000 7fe92dd9b700 1 --2- 172.21.15.49:0/2551002850 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe91802ff60 0x7fe918032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:44.527 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.525+0000 7fe92dd9b700 1 --2- 172.21.15.49:0/2551002850 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe91802ff60 0x7fe918032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe9240096a0 tx=0x7fe924006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:44.530 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.530+0000 7fe91effd700 1 -- 172.21.15.49:0/2551002850 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe920014420 con 0x7fe9300f6510 2023-11-30T21:48:44.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.746+0000 7fe934b6a700 1 -- 172.21.15.49:0/2551002850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe930108100 con 0x7fe9300f6510 2023-11-30T21:48:44.747 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.747+0000 7fe91effd700 1 -- 172.21.15.49:0/2551002850 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe92001d7f0 con 0x7fe9300f6510 2023-11-30T21:48:44.748 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:44.748 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:44.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.748+0000 7fe934b6a700 1 -- 172.21.15.49:0/2551002850 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe91802ff60 msgr2=0x7fe918032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:44.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.748+0000 7fe934b6a700 1 --2- 172.21.15.49:0/2551002850 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe91802ff60 0x7fe918032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fe9240096a0 tx=0x7fe924006b40).stop 2023-11-30T21:48:44.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.748+0000 7fe934b6a700 1 -- 172.21.15.49:0/2551002850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9300f6510 msgr2=0x7fe9301066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:44.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.748+0000 7fe934b6a700 1 --2- 172.21.15.49:0/2551002850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9300f6510 0x7fe9301066d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fe920013510 tx=0x7fe92000b040).stop 2023-11-30T21:48:44.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.748+0000 7fe934b6a700 1 -- 172.21.15.49:0/2551002850 shutdown_connections 2023-11-30T21:48:44.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.748+0000 7fe934b6a700 1 --2- 172.21.15.49:0/2551002850 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe91802ff60 0x7fe918032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:44.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.748+0000 7fe934b6a700 1 --2- 172.21.15.49:0/2551002850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9300f6510 0x7fe9301066d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:44.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.748+0000 7fe934b6a700 1 -- 172.21.15.49:0/2551002850 >> 172.21.15.49:0/2551002850 conn(0x7fe9300f1c70 msgr2=0x7fe9300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:44.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.748+0000 7fe934b6a700 1 -- 172.21.15.49:0/2551002850 shutdown_connections 2023-11-30T21:48:44.750 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:44.748+0000 7fe934b6a700 1 -- 172.21.15.49:0/2551002850 wait complete. 2023-11-30T21:48:44.750 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:45 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:43.501887+0000 mgr.smithi027.brrjcl (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:45 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:44.747374+0000 mon.smithi027 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.49:0/2551002850' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:46.141 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:46.142 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:47.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.461+0000 7fdad85d8700 1 -- 172.21.15.49:0/3146572831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad00f5410 msgr2=0x7fdad00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:47.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.461+0000 7fdad85d8700 1 --2- 172.21.15.49:0/3146572831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad00f5410 0x7fdad00f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fdac0009fb0 tx=0x7fdac0009b30).stop 2023-11-30T21:48:47.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.461+0000 7fdad85d8700 1 -- 172.21.15.49:0/3146572831 shutdown_connections 2023-11-30T21:48:47.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.461+0000 7fdad85d8700 1 --2- 172.21.15.49:0/3146572831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad00f5410 0x7fdad00f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:47.463 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.461+0000 7fdad85d8700 1 -- 172.21.15.49:0/3146572831 >> 172.21.15.49:0/3146572831 conn(0x7fdad00f0b70 msgr2=0x7fdad00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:47.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.461+0000 7fdad85d8700 1 -- 172.21.15.49:0/3146572831 shutdown_connections 2023-11-30T21:48:47.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.461+0000 7fdad85d8700 1 -- 172.21.15.49:0/3146572831 wait complete. 2023-11-30T21:48:47.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.462+0000 7fdad85d8700 1 Processor -- start 2023-11-30T21:48:47.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.462+0000 7fdad85d8700 1 -- start start 2023-11-30T21:48:47.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.462+0000 7fdad85d8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad00f5410 0x7fdad00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:47.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.462+0000 7fdad85d8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad00fb820 con 0x7fdad00f5410 2023-11-30T21:48:47.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.462+0000 7fdad6374700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad00f5410 0x7fdad00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:47.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.462+0000 7fdad6374700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad00f5410 0x7fdad00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:57234/0 (socket says 172.21.15.49:57234) 2023-11-30T21:48:47.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.462+0000 7fdad6374700 1 -- 172.21.15.49:0/1001230531 learned_addr learned my addr 172.21.15.49:0/1001230531 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:47.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.463+0000 7fdad6374700 1 -- 172.21.15.49:0/1001230531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac0005040 con 0x7fdad00f5410 2023-11-30T21:48:47.464 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.463+0000 7fdad6374700 1 --2- 172.21.15.49:0/1001230531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad00f5410 0x7fdad00fd3a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fdac0013510 tx=0x7fdac000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:47.465 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.463+0000 7fdac6ffd700 1 -- 172.21.15.49:0/1001230531 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdac0009320 con 0x7fdad00f5410 2023-11-30T21:48:47.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.463+0000 7fdac6ffd700 1 -- 172.21.15.49:0/1001230531 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdac0013f00 con 0x7fdad00f5410 2023-11-30T21:48:47.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.463+0000 7fdad85d8700 1 -- 172.21.15.49:0/1001230531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad00fb9c0 con 0x7fdad00f5410 2023-11-30T21:48:47.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.463+0000 7fdac6ffd700 1 -- 172.21.15.49:0/1001230531 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdac0014d90 con 0x7fdad00f5410 2023-11-30T21:48:47.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.463+0000 7fdad85d8700 1 -- 172.21.15.49:0/1001230531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad00fbe50 con 0x7fdad00f5410 2023-11-30T21:48:47.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.464+0000 7fdad85d8700 1 -- 172.21.15.49:0/1001230531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad0059100 con 0x7fdad00f5410 2023-11-30T21:48:47.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.464+0000 7fdac6ffd700 1 -- 172.21.15.49:0/1001230531 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdac00095f0 con 0x7fdad00f5410 2023-11-30T21:48:47.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.464+0000 7fdac6ffd700 1 --2- 172.21.15.49:0/1001230531 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdabc02ffb0 0x7fdabc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:47.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.464+0000 7fdac6ffd700 1 -- 172.21.15.49:0/1001230531 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdac0040fa0 con 0x7fdad00f5410 2023-11-30T21:48:47.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.464+0000 7fdad5b73700 1 --2- 172.21.15.49:0/1001230531 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdabc02ffb0 0x7fdabc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:47.466 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.465+0000 7fdad5b73700 1 --2- 172.21.15.49:0/1001230531 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdabc02ffb0 0x7fdabc032470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fdacc0096a0 tx=0x7fdacc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:47.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.469+0000 7fdac6ffd700 1 -- 172.21.15.49:0/1001230531 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdac006d2b0 con 0x7fdad00f5410 2023-11-30T21:48:47.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:47 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:45.502126+0000 mgr.smithi027.brrjcl (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:47.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.687+0000 7fdad85d8700 1 -- 172.21.15.49:0/1001230531 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdad0045ad0 con 0x7fdad00f5410 2023-11-30T21:48:47.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.687+0000 7fdac6ffd700 1 -- 172.21.15.49:0/1001230531 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdac0014420 con 0x7fdad00f5410 2023-11-30T21:48:47.688 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:47.688 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:47.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.689+0000 7fdad85d8700 1 -- 172.21.15.49:0/1001230531 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdabc02ffb0 msgr2=0x7fdabc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:47.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.689+0000 7fdad85d8700 1 --2- 172.21.15.49:0/1001230531 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdabc02ffb0 0x7fdabc032470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fdacc0096a0 tx=0x7fdacc006b40).stop 2023-11-30T21:48:47.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.689+0000 7fdad85d8700 1 -- 172.21.15.49:0/1001230531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad00f5410 msgr2=0x7fdad00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:47.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.689+0000 7fdad85d8700 1 --2- 172.21.15.49:0/1001230531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad00f5410 0x7fdad00fd3a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fdac0013510 tx=0x7fdac000b040).stop 2023-11-30T21:48:47.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.689+0000 7fdad85d8700 1 -- 172.21.15.49:0/1001230531 shutdown_connections 2023-11-30T21:48:47.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.689+0000 7fdad85d8700 1 --2- 172.21.15.49:0/1001230531 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdabc02ffb0 0x7fdabc032470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:47.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.689+0000 7fdad85d8700 1 --2- 172.21.15.49:0/1001230531 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdad00f5410 0x7fdad00fd3a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:47.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.689+0000 7fdad85d8700 1 -- 172.21.15.49:0/1001230531 >> 172.21.15.49:0/1001230531 conn(0x7fdad00f0b70 msgr2=0x7fdad005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:47.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.689+0000 7fdad85d8700 1 -- 172.21.15.49:0/1001230531 shutdown_connections 2023-11-30T21:48:47.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:47.689+0000 7fdad85d8700 1 -- 172.21.15.49:0/1001230531 wait complete. 2023-11-30T21:48:47.693 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:48.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:48 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:47.687975+0000 mon.smithi027 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.49:0/1001230531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:49.079 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:49.079 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:49.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:49 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:47.502495+0000 mgr.smithi027.brrjcl (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:50.332 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.327+0000 7efe83967700 1 -- 172.21.15.49:0/1063778105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7c0f6510 msgr2=0x7efe7c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:50.332 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.327+0000 7efe83967700 1 --2- 172.21.15.49:0/1063778105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7c0f6510 0x7efe7c0f6930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7efe6c009fb0 tx=0x7efe6c009b30).stop 2023-11-30T21:48:50.332 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.327+0000 7efe83967700 1 -- 172.21.15.49:0/1063778105 shutdown_connections 2023-11-30T21:48:50.332 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.327+0000 7efe83967700 1 --2- 172.21.15.49:0/1063778105 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7c0f6510 0x7efe7c0f6930 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.327+0000 7efe83967700 1 -- 172.21.15.49:0/1063778105 >> 172.21.15.49:0/1063778105 conn(0x7efe7c0f1c70 msgr2=0x7efe7c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.327+0000 7efe83967700 1 -- 172.21.15.49:0/1063778105 shutdown_connections 2023-11-30T21:48:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.327+0000 7efe83967700 1 -- 172.21.15.49:0/1063778105 wait complete. 2023-11-30T21:48:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.328+0000 7efe83967700 1 Processor -- start 2023-11-30T21:48:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.328+0000 7efe83967700 1 -- start start 2023-11-30T21:48:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.328+0000 7efe83967700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7c0f6510 0x7efe7c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.328+0000 7efe83967700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c106c20 con 0x7efe7c0f6510 2023-11-30T21:48:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.328+0000 7efe81703700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7c0f6510 0x7efe7c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.328+0000 7efe81703700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7c0f6510 0x7efe7c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:44942/0 (socket says 172.21.15.49:44942) 2023-11-30T21:48:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.328+0000 7efe81703700 1 -- 172.21.15.49:0/2660549613 learned_addr learned my addr 172.21.15.49:0/2660549613 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.328+0000 7efe81703700 1 -- 172.21.15.49:0/2660549613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c005040 con 0x7efe7c0f6510 2023-11-30T21:48:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.329+0000 7efe81703700 1 --2- 172.21.15.49:0/2660549613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7c0f6510 0x7efe7c106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7efe6c013510 tx=0x7efe6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.329+0000 7efe71ffb700 1 -- 172.21.15.49:0/2660549613 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe6c009320 con 0x7efe7c0f6510 2023-11-30T21:48:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.329+0000 7efe83967700 1 -- 172.21.15.49:0/2660549613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7c106e20 con 0x7efe7c0f6510 2023-11-30T21:48:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.329+0000 7efe71ffb700 1 -- 172.21.15.49:0/2660549613 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe6c013f00 con 0x7efe7c0f6510 2023-11-30T21:48:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.329+0000 7efe71ffb700 1 -- 172.21.15.49:0/2660549613 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe6c014d90 con 0x7efe7c0f6510 2023-11-30T21:48:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.329+0000 7efe83967700 1 -- 172.21.15.49:0/2660549613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7c1072b0 con 0x7efe7c0f6510 2023-11-30T21:48:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.330+0000 7efe71ffb700 1 -- 172.21.15.49:0/2660549613 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe6c0095f0 con 0x7efe7c0f6510 2023-11-30T21:48:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.330+0000 7efe83967700 1 -- 172.21.15.49:0/2660549613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7c045ad0 con 0x7efe7c0f6510 2023-11-30T21:48:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.330+0000 7efe71ffb700 1 --2- 172.21.15.49:0/2660549613 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efe6802ff60 0x7efe68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.330+0000 7efe71ffb700 1 -- 172.21.15.49:0/2660549613 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe6c0410c0 con 0x7efe7c0f6510 2023-11-30T21:48:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.330+0000 7efe80f02700 1 --2- 172.21.15.49:0/2660549613 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efe6802ff60 0x7efe68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:50.334 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.331+0000 7efe80f02700 1 --2- 172.21.15.49:0/2660549613 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efe6802ff60 0x7efe68032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7efe780096a0 tx=0x7efe78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:50.335 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.335+0000 7efe71ffb700 1 -- 172.21.15.49:0/2660549613 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe6c01d7f0 con 0x7efe7c0f6510 2023-11-30T21:48:50.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:50 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:49.502860+0000 mgr.smithi027.brrjcl (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:50.553 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.553+0000 7efe83967700 1 -- 172.21.15.49:0/2660549613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe7c0fbf60 con 0x7efe7c0f6510 2023-11-30T21:48:50.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.553+0000 7efe71ffb700 1 -- 172.21.15.49:0/2660549613 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe6c010230 con 0x7efe7c0f6510 2023-11-30T21:48:50.554 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:50.555 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:50.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.555+0000 7efe83967700 1 -- 172.21.15.49:0/2660549613 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efe6802ff60 msgr2=0x7efe68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:50.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.555+0000 7efe83967700 1 --2- 172.21.15.49:0/2660549613 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efe6802ff60 0x7efe68032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7efe780096a0 tx=0x7efe78006b40).stop 2023-11-30T21:48:50.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.555+0000 7efe83967700 1 -- 172.21.15.49:0/2660549613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7c0f6510 msgr2=0x7efe7c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:50.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.555+0000 7efe83967700 1 --2- 172.21.15.49:0/2660549613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7c0f6510 0x7efe7c106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7efe6c013510 tx=0x7efe6c00b040).stop 2023-11-30T21:48:50.556 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.555+0000 7efe83967700 1 -- 172.21.15.49:0/2660549613 shutdown_connections 2023-11-30T21:48:50.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.555+0000 7efe83967700 1 --2- 172.21.15.49:0/2660549613 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efe6802ff60 0x7efe68032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:50.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.555+0000 7efe83967700 1 --2- 172.21.15.49:0/2660549613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7c0f6510 0x7efe7c106730 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:50.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.555+0000 7efe83967700 1 -- 172.21.15.49:0/2660549613 >> 172.21.15.49:0/2660549613 conn(0x7efe7c0f1c70 msgr2=0x7efe7c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:50.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.555+0000 7efe83967700 1 -- 172.21.15.49:0/2660549613 shutdown_connections 2023-11-30T21:48:50.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:50.555+0000 7efe83967700 1 -- 172.21.15.49:0/2660549613 wait complete. 2023-11-30T21:48:50.557 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:51.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:50.553763+0000 mon.smithi027 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.49:0/2660549613' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:51.947 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:51.947 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:52.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:52 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:51.503254+0000 mgr.smithi027.brrjcl (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:53.181 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.179+0000 7f59cc452700 1 -- 172.21.15.49:0/2139589061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59c40f5410 msgr2=0x7f59c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:53.182 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.179+0000 7f59cc452700 1 --2- 172.21.15.49:0/2139589061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59c40f5410 0x7f59c40f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f59c0009fb0 tx=0x7f59c0009b30).stop 2023-11-30T21:48:53.182 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.180+0000 7f59cc452700 1 -- 172.21.15.49:0/2139589061 shutdown_connections 2023-11-30T21:48:53.182 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.180+0000 7f59cc452700 1 --2- 172.21.15.49:0/2139589061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59c40f5410 0x7f59c40f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:53.182 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.180+0000 7f59cc452700 1 -- 172.21.15.49:0/2139589061 >> 172.21.15.49:0/2139589061 conn(0x7f59c40f0b70 msgr2=0x7f59c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:53.182 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.180+0000 7f59cc452700 1 -- 172.21.15.49:0/2139589061 shutdown_connections 2023-11-30T21:48:53.182 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.180+0000 7f59cc452700 1 -- 172.21.15.49:0/2139589061 wait complete. 2023-11-30T21:48:53.182 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.180+0000 7f59cc452700 1 Processor -- start 2023-11-30T21:48:53.182 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.180+0000 7f59cc452700 1 -- start start 2023-11-30T21:48:53.182 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.180+0000 7f59cc452700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59c40f5410 0x7f59c4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:53.182 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.180+0000 7f59cc452700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c4060240 con 0x7f59c40f5410 2023-11-30T21:48:53.182 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.181+0000 7f59ca1ee700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59c40f5410 0x7f59c4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:53.183 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.181+0000 7f59ca1ee700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59c40f5410 0x7f59c4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:44968/0 (socket says 172.21.15.49:44968) 2023-11-30T21:48:53.183 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.181+0000 7f59ca1ee700 1 -- 172.21.15.49:0/1100878176 learned_addr learned my addr 172.21.15.49:0/1100878176 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:53.183 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.181+0000 7f59ca1ee700 1 -- 172.21.15.49:0/1100878176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c0005040 con 0x7f59c40f5410 2023-11-30T21:48:53.183 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.181+0000 7f59ca1ee700 1 --2- 172.21.15.49:0/1100878176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59c40f5410 0x7f59c4109430 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f59c0013510 tx=0x7f59c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:53.184 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.181+0000 7f59b6ffd700 1 -- 172.21.15.49:0/1100878176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59c0009320 con 0x7f59c40f5410 2023-11-30T21:48:53.184 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.181+0000 7f59cc452700 1 -- 172.21.15.49:0/1100878176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c4060440 con 0x7f59c40f5410 2023-11-30T21:48:53.184 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.181+0000 7f59cc452700 1 -- 172.21.15.49:0/1100878176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c4109ae0 con 0x7f59c40f5410 2023-11-30T21:48:53.184 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.181+0000 7f59b6ffd700 1 -- 172.21.15.49:0/1100878176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f59c0013f00 con 0x7f59c40f5410 2023-11-30T21:48:53.184 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.182+0000 7f59b6ffd700 1 -- 172.21.15.49:0/1100878176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59c0014d60 con 0x7f59c40f5410 2023-11-30T21:48:53.184 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.182+0000 7f59b6ffd700 1 -- 172.21.15.49:0/1100878176 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59c00095f0 con 0x7f59c40f5410 2023-11-30T21:48:53.184 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.182+0000 7f59cc452700 1 -- 172.21.15.49:0/1100878176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c4045ad0 con 0x7f59c40f5410 2023-11-30T21:48:53.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.182+0000 7f59b6ffd700 1 --2- 172.21.15.49:0/1100878176 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f59b002ff60 0x7f59b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:53.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.182+0000 7f59b6ffd700 1 -- 172.21.15.49:0/1100878176 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59c0041010 con 0x7f59c40f5410 2023-11-30T21:48:53.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.183+0000 7f59c99ed700 1 --2- 172.21.15.49:0/1100878176 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f59b002ff60 0x7f59b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:53.185 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.183+0000 7f59c99ed700 1 --2- 172.21.15.49:0/1100878176 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f59b002ff60 0x7f59b0032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f59b80096a0 tx=0x7f59b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:53.187 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.187+0000 7f59b6ffd700 1 -- 172.21.15.49:0/1100878176 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59c0014420 con 0x7f59c40f5410 2023-11-30T21:48:53.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.404+0000 7f59cc452700 1 -- 172.21.15.49:0/1100878176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59c40fae60 con 0x7f59c40f5410 2023-11-30T21:48:53.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.405+0000 7f59b6ffd700 1 -- 172.21.15.49:0/1100878176 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f59c0012980 con 0x7f59c40f5410 2023-11-30T21:48:53.405 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:53.406 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:53.407 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.406+0000 7f59cc452700 1 -- 172.21.15.49:0/1100878176 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f59b002ff60 msgr2=0x7f59b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:53.407 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.406+0000 7f59cc452700 1 --2- 172.21.15.49:0/1100878176 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f59b002ff60 0x7f59b0032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f59b80096a0 tx=0x7f59b8006b40).stop 2023-11-30T21:48:53.407 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.406+0000 7f59cc452700 1 -- 172.21.15.49:0/1100878176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59c40f5410 msgr2=0x7f59c4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:53.407 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.406+0000 7f59cc452700 1 --2- 172.21.15.49:0/1100878176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59c40f5410 0x7f59c4109430 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f59c0013510 tx=0x7f59c000b040).stop 2023-11-30T21:48:53.407 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.406+0000 7f59cc452700 1 -- 172.21.15.49:0/1100878176 shutdown_connections 2023-11-30T21:48:53.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.406+0000 7f59cc452700 1 --2- 172.21.15.49:0/1100878176 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f59b002ff60 0x7f59b0032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:53.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.406+0000 7f59cc452700 1 --2- 172.21.15.49:0/1100878176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59c40f5410 0x7f59c4109430 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:53.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.406+0000 7f59cc452700 1 -- 172.21.15.49:0/1100878176 >> 172.21.15.49:0/1100878176 conn(0x7f59c40f0b70 msgr2=0x7f59c40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:53.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.406+0000 7f59cc452700 1 -- 172.21.15.49:0/1100878176 shutdown_connections 2023-11-30T21:48:53.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:53.406+0000 7f59cc452700 1 -- 172.21.15.49:0/1100878176 wait complete. 2023-11-30T21:48:53.408 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:53.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:53 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:53.405305+0000 mon.smithi027 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.49:0/1100878176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:54.814 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:54.814 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:55.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:53.503585+0000 mgr.smithi027.brrjcl (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:56.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.011+0000 7f34f6903700 1 -- 172.21.15.49:0/3562525434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00f6510 msgr2=0x7f34f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:56.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.011+0000 7f34f6903700 1 --2- 172.21.15.49:0/3562525434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00f6510 0x7f34f00f6930 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f34e0009fb0 tx=0x7f34e0009b30).stop 2023-11-30T21:48:56.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.011+0000 7f34f6903700 1 -- 172.21.15.49:0/3562525434 shutdown_connections 2023-11-30T21:48:56.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.011+0000 7f34f6903700 1 --2- 172.21.15.49:0/3562525434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00f6510 0x7f34f00f6930 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:56.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.011+0000 7f34f6903700 1 -- 172.21.15.49:0/3562525434 >> 172.21.15.49:0/3562525434 conn(0x7f34f00f1c70 msgr2=0x7f34f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:56.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.011+0000 7f34f6903700 1 -- 172.21.15.49:0/3562525434 shutdown_connections 2023-11-30T21:48:56.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.011+0000 7f34f6903700 1 -- 172.21.15.49:0/3562525434 wait complete. 2023-11-30T21:48:56.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.012+0000 7f34f6903700 1 Processor -- start 2023-11-30T21:48:56.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.012+0000 7f34f6903700 1 -- start start 2023-11-30T21:48:56.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.012+0000 7f34f6903700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00f6510 0x7f34f0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:56.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.012+0000 7f34f6903700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f0106c20 con 0x7f34f00f6510 2023-11-30T21:48:56.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.012+0000 7f34effff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00f6510 0x7f34f0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:56.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.012+0000 7f34effff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00f6510 0x7f34f0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:44972/0 (socket says 172.21.15.49:44972) 2023-11-30T21:48:56.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.012+0000 7f34effff700 1 -- 172.21.15.49:0/731707819 learned_addr learned my addr 172.21.15.49:0/731707819 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:56.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.013+0000 7f34effff700 1 -- 172.21.15.49:0/731707819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e0005040 con 0x7f34f00f6510 2023-11-30T21:48:56.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.013+0000 7f34effff700 1 --2- 172.21.15.49:0/731707819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00f6510 0x7f34f0106730 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f34e0013510 tx=0x7f34e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:56.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.013+0000 7f34ecff9700 1 -- 172.21.15.49:0/731707819 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e0009320 con 0x7f34f00f6510 2023-11-30T21:48:56.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.013+0000 7f34ecff9700 1 -- 172.21.15.49:0/731707819 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34e0013f00 con 0x7f34f00f6510 2023-11-30T21:48:56.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.013+0000 7f34f6903700 1 -- 172.21.15.49:0/731707819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f0106e20 con 0x7f34f00f6510 2023-11-30T21:48:56.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.013+0000 7f34ecff9700 1 -- 172.21.15.49:0/731707819 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e0014d90 con 0x7f34f00f6510 2023-11-30T21:48:56.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.013+0000 7f34f6903700 1 -- 172.21.15.49:0/731707819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f01072b0 con 0x7f34f00f6510 2023-11-30T21:48:56.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.014+0000 7f34ecff9700 1 -- 172.21.15.49:0/731707819 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34e00095f0 con 0x7f34f00f6510 2023-11-30T21:48:56.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.014+0000 7f34f6903700 1 -- 172.21.15.49:0/731707819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f0045ad0 con 0x7f34f00f6510 2023-11-30T21:48:56.017 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.014+0000 7f34ecff9700 1 --2- 172.21.15.49:0/731707819 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f34dc02ff60 0x7f34dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:56.017 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.014+0000 7f34ecff9700 1 -- 172.21.15.49:0/731707819 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34e00410a0 con 0x7f34f00f6510 2023-11-30T21:48:56.017 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.014+0000 7f34ef7fe700 1 --2- 172.21.15.49:0/731707819 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f34dc02ff60 0x7f34dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:56.017 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.015+0000 7f34ef7fe700 1 --2- 172.21.15.49:0/731707819 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f34dc02ff60 0x7f34dc032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f34d80096a0 tx=0x7f34d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:56.020 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.019+0000 7f34ecff9700 1 -- 172.21.15.49:0/731707819 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34e001d7f0 con 0x7f34f00f6510 2023-11-30T21:48:56.238 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.237+0000 7f34f6903700 1 -- 172.21.15.49:0/731707819 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34f00fbf60 con 0x7f34f00f6510 2023-11-30T21:48:56.239 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.237+0000 7f34ecff9700 1 -- 172.21.15.49:0/731707819 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f34e0010230 con 0x7f34f00f6510 2023-11-30T21:48:56.239 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:56.239 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:56.240 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.239+0000 7f34f6903700 1 -- 172.21.15.49:0/731707819 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f34dc02ff60 msgr2=0x7f34dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:56.240 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.239+0000 7f34f6903700 1 --2- 172.21.15.49:0/731707819 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f34dc02ff60 0x7f34dc032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f34d80096a0 tx=0x7f34d8006b40).stop 2023-11-30T21:48:56.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.239+0000 7f34f6903700 1 -- 172.21.15.49:0/731707819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00f6510 msgr2=0x7f34f0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:56.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.239+0000 7f34f6903700 1 --2- 172.21.15.49:0/731707819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00f6510 0x7f34f0106730 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f34e0013510 tx=0x7f34e000b040).stop 2023-11-30T21:48:56.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.239+0000 7f34f6903700 1 -- 172.21.15.49:0/731707819 shutdown_connections 2023-11-30T21:48:56.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.239+0000 7f34f6903700 1 --2- 172.21.15.49:0/731707819 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f34dc02ff60 0x7f34dc032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:56.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.239+0000 7f34f6903700 1 --2- 172.21.15.49:0/731707819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00f6510 0x7f34f0106730 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:56.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.239+0000 7f34f6903700 1 -- 172.21.15.49:0/731707819 >> 172.21.15.49:0/731707819 conn(0x7f34f00f1c70 msgr2=0x7f34f00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:56.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.239+0000 7f34f6903700 1 -- 172.21.15.49:0/731707819 shutdown_connections 2023-11-30T21:48:56.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:56.239+0000 7f34f6903700 1 -- 172.21.15.49:0/731707819 wait complete. 2023-11-30T21:48:56.241 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:57.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:57 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:55.503959+0000 mgr.smithi027.brrjcl (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:48:57.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:57 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:56.238937+0000 mon.smithi027 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.49:0/731707819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:48:57.636 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:48:57.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:48:58.872 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.870+0000 7f5da48ba700 1 -- 172.21.15.49:0/2193206024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5da00f5410 msgr2=0x7f5da00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:58.872 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.870+0000 7f5da48ba700 1 --2- 172.21.15.49:0/2193206024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5da00f5410 0x7f5da00f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f5d94009fb0 tx=0x7f5d94009b30).stop 2023-11-30T21:48:58.872 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.870+0000 7f5da48ba700 1 -- 172.21.15.49:0/2193206024 shutdown_connections 2023-11-30T21:48:58.873 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.870+0000 7f5da48ba700 1 --2- 172.21.15.49:0/2193206024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5da00f5410 0x7f5da00f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:58.873 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.870+0000 7f5da48ba700 1 -- 172.21.15.49:0/2193206024 >> 172.21.15.49:0/2193206024 conn(0x7f5da00f0b70 msgr2=0x7f5da00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:58.873 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.870+0000 7f5da48ba700 1 -- 172.21.15.49:0/2193206024 shutdown_connections 2023-11-30T21:48:58.873 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.870+0000 7f5da48ba700 1 -- 172.21.15.49:0/2193206024 wait complete. 2023-11-30T21:48:58.873 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.871+0000 7f5da48ba700 1 Processor -- start 2023-11-30T21:48:58.873 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.871+0000 7f5da48ba700 1 -- start start 2023-11-30T21:48:58.875 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.871+0000 7f5da48ba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5da00f5410 0x7f5da0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:58.875 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.871+0000 7f5da48ba700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da0105b20 con 0x7f5da00f5410 2023-11-30T21:48:58.875 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.871+0000 7f5d9e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5da00f5410 0x7f5da0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:58.875 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.871+0000 7f5d9e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5da00f5410 0x7f5da0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:41230/0 (socket says 172.21.15.49:41230) 2023-11-30T21:48:58.875 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.871+0000 7f5d9e59c700 1 -- 172.21.15.49:0/1898173680 learned_addr learned my addr 172.21.15.49:0/1898173680 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:48:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.872+0000 7f5d9e59c700 1 -- 172.21.15.49:0/1898173680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d94005040 con 0x7f5da00f5410 2023-11-30T21:48:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.872+0000 7f5d9e59c700 1 --2- 172.21.15.49:0/1898173680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5da00f5410 0x7f5da0105630 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f5d94013510 tx=0x7f5d9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.872+0000 7f5d8effd700 1 -- 172.21.15.49:0/1898173680 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d94009320 con 0x7f5da00f5410 2023-11-30T21:48:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.872+0000 7f5da48ba700 1 -- 172.21.15.49:0/1898173680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da0105d20 con 0x7f5da00f5410 2023-11-30T21:48:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.872+0000 7f5d8effd700 1 -- 172.21.15.49:0/1898173680 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d94013f00 con 0x7f5da00f5410 2023-11-30T21:48:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.872+0000 7f5d8effd700 1 -- 172.21.15.49:0/1898173680 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d94014d90 con 0x7f5da00f5410 2023-11-30T21:48:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.872+0000 7f5da48ba700 1 -- 172.21.15.49:0/1898173680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da01061b0 con 0x7f5da00f5410 2023-11-30T21:48:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.873+0000 7f5da48ba700 1 -- 172.21.15.49:0/1898173680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da0045ad0 con 0x7f5da00f5410 2023-11-30T21:48:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.873+0000 7f5d8effd700 1 -- 172.21.15.49:0/1898173680 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d940095f0 con 0x7f5da00f5410 2023-11-30T21:48:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.873+0000 7f5d8effd700 1 --2- 172.21.15.49:0/1898173680 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5d8802ff60 0x7f5d88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:48:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.873+0000 7f5d8effd700 1 -- 172.21.15.49:0/1898173680 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d94041000 con 0x7f5da00f5410 2023-11-30T21:48:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.873+0000 7f5d9dd9b700 1 --2- 172.21.15.49:0/1898173680 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5d8802ff60 0x7f5d88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:48:58.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.874+0000 7f5d9dd9b700 1 --2- 172.21.15.49:0/1898173680 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5d8802ff60 0x7f5d88032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f5d900096a0 tx=0x7f5d90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:48:58.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:58.877+0000 7f5d8effd700 1 -- 172.21.15.49:0/1898173680 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d94014420 con 0x7f5da00f5410 2023-11-30T21:48:59.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:59.097+0000 7f5da48ba700 1 -- 172.21.15.49:0/1898173680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5da00fae60 con 0x7f5da00f5410 2023-11-30T21:48:59.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:59.097+0000 7f5d8effd700 1 -- 172.21.15.49:0/1898173680 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d94010230 con 0x7f5da00f5410 2023-11-30T21:48:59.099 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:48:59.099 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:48:59.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:59.099+0000 7f5da48ba700 1 -- 172.21.15.49:0/1898173680 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5d8802ff60 msgr2=0x7f5d88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:59.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:59.099+0000 7f5da48ba700 1 --2- 172.21.15.49:0/1898173680 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5d8802ff60 0x7f5d88032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f5d900096a0 tx=0x7f5d90006b40).stop 2023-11-30T21:48:59.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:59.099+0000 7f5da48ba700 1 -- 172.21.15.49:0/1898173680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5da00f5410 msgr2=0x7f5da0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:48:59.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:59.099+0000 7f5da48ba700 1 --2- 172.21.15.49:0/1898173680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5da00f5410 0x7f5da0105630 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f5d94013510 tx=0x7f5d9400b040).stop 2023-11-30T21:48:59.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:59.099+0000 7f5da48ba700 1 -- 172.21.15.49:0/1898173680 shutdown_connections 2023-11-30T21:48:59.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:59.099+0000 7f5da48ba700 1 --2- 172.21.15.49:0/1898173680 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5d8802ff60 0x7f5d88032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:59.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:59.099+0000 7f5da48ba700 1 --2- 172.21.15.49:0/1898173680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5da00f5410 0x7f5da0105630 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:48:59.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:59.099+0000 7f5da48ba700 1 -- 172.21.15.49:0/1898173680 >> 172.21.15.49:0/1898173680 conn(0x7f5da00f0b70 msgr2=0x7f5da00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:48:59.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:59.099+0000 7f5da48ba700 1 -- 172.21.15.49:0/1898173680 shutdown_connections 2023-11-30T21:48:59.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:48:59.099+0000 7f5da48ba700 1 -- 172.21.15.49:0/1898173680 wait complete. 2023-11-30T21:48:59.102 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:48:59.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:48:59 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:57.504283+0000 mgr.smithi027.brrjcl (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:00.483 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:00.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:00.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:00 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:48:59.099124+0000 mon.smithi027 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.49:0/1898173680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:01.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:01 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:48:59.504735+0000 mgr.smithi027.brrjcl (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:01.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.689+0000 7f654b6fe700 1 -- 172.21.15.49:0/3807822271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65440f6510 msgr2=0x7f65440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:01.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.689+0000 7f654b6fe700 1 --2- 172.21.15.49:0/3807822271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65440f6510 0x7f65440f6930 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f6540009fb0 tx=0x7f6540009b30).stop 2023-11-30T21:49:01.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.689+0000 7f654b6fe700 1 -- 172.21.15.49:0/3807822271 shutdown_connections 2023-11-30T21:49:01.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.689+0000 7f654b6fe700 1 --2- 172.21.15.49:0/3807822271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65440f6510 0x7f65440f6930 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:01.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.689+0000 7f654b6fe700 1 -- 172.21.15.49:0/3807822271 >> 172.21.15.49:0/3807822271 conn(0x7f65440f1c70 msgr2=0x7f65440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:01.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.689+0000 7f654b6fe700 1 -- 172.21.15.49:0/3807822271 shutdown_connections 2023-11-30T21:49:01.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.689+0000 7f654b6fe700 1 -- 172.21.15.49:0/3807822271 wait complete. 2023-11-30T21:49:01.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.690+0000 7f654b6fe700 1 Processor -- start 2023-11-30T21:49:01.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.690+0000 7f654b6fe700 1 -- start start 2023-11-30T21:49:01.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.690+0000 7f654b6fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65440f6510 0x7f6544106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:01.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.690+0000 7f654b6fe700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6544106c20 con 0x7f65440f6510 2023-11-30T21:49:01.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.690+0000 7f654949a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65440f6510 0x7f6544106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:01.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.690+0000 7f654949a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65440f6510 0x7f6544106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:41248/0 (socket says 172.21.15.49:41248) 2023-11-30T21:49:01.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.690+0000 7f654949a700 1 -- 172.21.15.49:0/4049496621 learned_addr learned my addr 172.21.15.49:0/4049496621 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:01.693 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.690+0000 7f654949a700 1 -- 172.21.15.49:0/4049496621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6540005040 con 0x7f65440f6510 2023-11-30T21:49:01.694 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.691+0000 7f654949a700 1 --2- 172.21.15.49:0/4049496621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65440f6510 0x7f6544106730 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f6540013510 tx=0x7f654000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:01.694 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.691+0000 7f6535ffb700 1 -- 172.21.15.49:0/4049496621 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6540009320 con 0x7f65440f6510 2023-11-30T21:49:01.694 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.691+0000 7f6535ffb700 1 -- 172.21.15.49:0/4049496621 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6540013f00 con 0x7f65440f6510 2023-11-30T21:49:01.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.691+0000 7f654b6fe700 1 -- 172.21.15.49:0/4049496621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6544106e20 con 0x7f65440f6510 2023-11-30T21:49:01.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.691+0000 7f6535ffb700 1 -- 172.21.15.49:0/4049496621 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6540014d90 con 0x7f65440f6510 2023-11-30T21:49:01.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.691+0000 7f654b6fe700 1 -- 172.21.15.49:0/4049496621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65441072b0 con 0x7f65440f6510 2023-11-30T21:49:01.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.692+0000 7f6535ffb700 1 -- 172.21.15.49:0/4049496621 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f65400095f0 con 0x7f65440f6510 2023-11-30T21:49:01.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.692+0000 7f654b6fe700 1 -- 172.21.15.49:0/4049496621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6544045ad0 con 0x7f65440f6510 2023-11-30T21:49:01.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.692+0000 7f6535ffb700 1 --2- 172.21.15.49:0/4049496621 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f653002ff60 0x7f6530032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:01.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.692+0000 7f6535ffb700 1 -- 172.21.15.49:0/4049496621 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6540041000 con 0x7f65440f6510 2023-11-30T21:49:01.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.692+0000 7f6548c99700 1 --2- 172.21.15.49:0/4049496621 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f653002ff60 0x7f6530032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:01.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.693+0000 7f6548c99700 1 --2- 172.21.15.49:0/4049496621 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f653002ff60 0x7f6530032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f65380096a0 tx=0x7f6538006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:01.698 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.697+0000 7f6535ffb700 1 -- 172.21.15.49:0/4049496621 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6540014420 con 0x7f65440f6510 2023-11-30T21:49:01.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.916+0000 7f654b6fe700 1 -- 172.21.15.49:0/4049496621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65440fbf60 con 0x7f65440f6510 2023-11-30T21:49:01.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.916+0000 7f6535ffb700 1 -- 172.21.15.49:0/4049496621 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f654001d7f0 con 0x7f65440f6510 2023-11-30T21:49:01.918 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:01.918 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:01.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.918+0000 7f654b6fe700 1 -- 172.21.15.49:0/4049496621 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f653002ff60 msgr2=0x7f6530032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:01.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.918+0000 7f654b6fe700 1 --2- 172.21.15.49:0/4049496621 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f653002ff60 0x7f6530032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f65380096a0 tx=0x7f6538006b40).stop 2023-11-30T21:49:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.918+0000 7f654b6fe700 1 -- 172.21.15.49:0/4049496621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65440f6510 msgr2=0x7f6544106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.918+0000 7f654b6fe700 1 --2- 172.21.15.49:0/4049496621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65440f6510 0x7f6544106730 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f6540013510 tx=0x7f654000b040).stop 2023-11-30T21:49:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.918+0000 7f654b6fe700 1 -- 172.21.15.49:0/4049496621 shutdown_connections 2023-11-30T21:49:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.918+0000 7f654b6fe700 1 --2- 172.21.15.49:0/4049496621 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f653002ff60 0x7f6530032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.918+0000 7f654b6fe700 1 --2- 172.21.15.49:0/4049496621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f65440f6510 0x7f6544106730 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.918+0000 7f654b6fe700 1 -- 172.21.15.49:0/4049496621 >> 172.21.15.49:0/4049496621 conn(0x7f65440f1c70 msgr2=0x7f65440f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.918+0000 7f654b6fe700 1 -- 172.21.15.49:0/4049496621 shutdown_connections 2023-11-30T21:49:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:01.918+0000 7f654b6fe700 1 -- 172.21.15.49:0/4049496621 wait complete. 2023-11-30T21:49:01.920 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:02.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:02 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:01.918111+0000 mon.smithi027 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.49:0/4049496621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:03.309 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:03.309 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:03.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:03 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:01.505213+0000 mgr.smithi027.brrjcl (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:03.587825+0000 mon.smithi027 (mon.0) 215 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/mirror_snapshot_schedule"}]: dispatch 2023-11-30T21:49:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:03.594781+0000 mon.smithi027 (mon.0) 216 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/trash_purge_schedule"}]: dispatch 2023-11-30T21:49:04.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.515+0000 7f9f3ce24700 1 -- 172.21.15.49:0/2397517905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f380f5410 msgr2=0x7f9f380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:04.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.515+0000 7f9f3ce24700 1 --2- 172.21.15.49:0/2397517905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f380f5410 0x7f9f380f5830 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f9f28009fb0 tx=0x7f9f28009b30).stop 2023-11-30T21:49:04.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.515+0000 7f9f3ce24700 1 -- 172.21.15.49:0/2397517905 shutdown_connections 2023-11-30T21:49:04.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.515+0000 7f9f3ce24700 1 --2- 172.21.15.49:0/2397517905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f380f5410 0x7f9f380f5830 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:04.518 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.515+0000 7f9f3ce24700 1 -- 172.21.15.49:0/2397517905 >> 172.21.15.49:0/2397517905 conn(0x7f9f380f0b70 msgr2=0x7f9f380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.515+0000 7f9f3ce24700 1 -- 172.21.15.49:0/2397517905 shutdown_connections 2023-11-30T21:49:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.515+0000 7f9f3ce24700 1 -- 172.21.15.49:0/2397517905 wait complete. 2023-11-30T21:49:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.515+0000 7f9f3ce24700 1 Processor -- start 2023-11-30T21:49:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.515+0000 7f9f3ce24700 1 -- start start 2023-11-30T21:49:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.516+0000 7f9f3ce24700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f380f5410 0x7f9f38109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.516+0000 7f9f3ce24700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f38060240 con 0x7f9f380f5410 2023-11-30T21:49:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.516+0000 7f9f3659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f380f5410 0x7f9f38109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.516+0000 7f9f3659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f380f5410 0x7f9f38109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:41278/0 (socket says 172.21.15.49:41278) 2023-11-30T21:49:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.516+0000 7f9f3659c700 1 -- 172.21.15.49:0/1636366700 learned_addr learned my addr 172.21.15.49:0/1636366700 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.516+0000 7f9f3659c700 1 -- 172.21.15.49:0/1636366700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f28005040 con 0x7f9f380f5410 2023-11-30T21:49:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.516+0000 7f9f3659c700 1 --2- 172.21.15.49:0/1636366700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f380f5410 0x7f9f38109430 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f9f28013510 tx=0x7f9f2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.516+0000 7f9f26ffd700 1 -- 172.21.15.49:0/1636366700 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f28009320 con 0x7f9f380f5410 2023-11-30T21:49:04.519 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.516+0000 7f9f3ce24700 1 -- 172.21.15.49:0/1636366700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f38060440 con 0x7f9f380f5410 2023-11-30T21:49:04.520 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.516+0000 7f9f26ffd700 1 -- 172.21.15.49:0/1636366700 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f28013f00 con 0x7f9f380f5410 2023-11-30T21:49:04.520 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.516+0000 7f9f26ffd700 1 -- 172.21.15.49:0/1636366700 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f28014d90 con 0x7f9f380f5410 2023-11-30T21:49:04.520 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.516+0000 7f9f3ce24700 1 -- 172.21.15.49:0/1636366700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f38109ae0 con 0x7f9f380f5410 2023-11-30T21:49:04.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.517+0000 7f9f26ffd700 1 -- 172.21.15.49:0/1636366700 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9f280263a0 con 0x7f9f380f5410 2023-11-30T21:49:04.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.517+0000 7f9f26ffd700 1 --2- 172.21.15.49:0/1636366700 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9f2002ff60 0x7f9f20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:04.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.517+0000 7f9f26ffd700 1 -- 172.21.15.49:0/1636366700 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9f28041040 con 0x7f9f380f5410 2023-11-30T21:49:04.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.517+0000 7f9f3ce24700 1 -- 172.21.15.49:0/1636366700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f38045ad0 con 0x7f9f380f5410 2023-11-30T21:49:04.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.518+0000 7f9f35d9b700 1 --2- 172.21.15.49:0/1636366700 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9f2002ff60 0x7f9f20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:04.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.518+0000 7f9f35d9b700 1 --2- 172.21.15.49:0/1636366700 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9f2002ff60 0x7f9f20032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9f2c0096a0 tx=0x7f9f2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:04.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.522+0000 7f9f26ffd700 1 -- 172.21.15.49:0/1636366700 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9f28012920 con 0x7f9f380f5410 2023-11-30T21:49:04.743 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.741+0000 7f9f3ce24700 1 -- 172.21.15.49:0/1636366700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f3810a1f0 con 0x7f9f380f5410 2023-11-30T21:49:04.744 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.742+0000 7f9f26ffd700 1 -- 172.21.15.49:0/1636366700 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9f2802b020 con 0x7f9f380f5410 2023-11-30T21:49:04.744 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:04.744 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:04.745 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.743+0000 7f9f3ce24700 1 -- 172.21.15.49:0/1636366700 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9f2002ff60 msgr2=0x7f9f20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:04.745 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.743+0000 7f9f3ce24700 1 --2- 172.21.15.49:0/1636366700 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9f2002ff60 0x7f9f20032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9f2c0096a0 tx=0x7f9f2c006b40).stop 2023-11-30T21:49:04.745 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.743+0000 7f9f3ce24700 1 -- 172.21.15.49:0/1636366700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f380f5410 msgr2=0x7f9f38109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:04.745 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.743+0000 7f9f3ce24700 1 --2- 172.21.15.49:0/1636366700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f380f5410 0x7f9f38109430 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f9f28013510 tx=0x7f9f2800b040).stop 2023-11-30T21:49:04.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.744+0000 7f9f3ce24700 1 -- 172.21.15.49:0/1636366700 shutdown_connections 2023-11-30T21:49:04.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.744+0000 7f9f3ce24700 1 --2- 172.21.15.49:0/1636366700 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9f2002ff60 0x7f9f20032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:04.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.744+0000 7f9f3ce24700 1 --2- 172.21.15.49:0/1636366700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f380f5410 0x7f9f38109430 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:04.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.744+0000 7f9f3ce24700 1 -- 172.21.15.49:0/1636366700 >> 172.21.15.49:0/1636366700 conn(0x7f9f380f0b70 msgr2=0x7f9f380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:04.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.744+0000 7f9f3ce24700 1 -- 172.21.15.49:0/1636366700 shutdown_connections 2023-11-30T21:49:04.746 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:04.744+0000 7f9f3ce24700 1 -- 172.21.15.49:0/1636366700 wait complete. 2023-11-30T21:49:04.746 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:05.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:03.505614+0000 mgr.smithi027.brrjcl (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:05.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:04.743391+0000 mon.smithi027 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.49:0/1636366700' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:06.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:06.129 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:07.379 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.377+0000 7f15db034700 1 -- 172.21.15.49:0/1661364508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15d40f78e0 msgr2=0x7f15d40f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:07.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.377+0000 7f15db034700 1 --2- 172.21.15.49:0/1661364508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15d40f78e0 0x7f15d40f7d00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f15d0009fb0 tx=0x7f15d0009b30).stop 2023-11-30T21:49:07.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.377+0000 7f15db034700 1 -- 172.21.15.49:0/1661364508 shutdown_connections 2023-11-30T21:49:07.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.377+0000 7f15db034700 1 --2- 172.21.15.49:0/1661364508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15d40f78e0 0x7f15d40f7d00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:07.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.377+0000 7f15db034700 1 -- 172.21.15.49:0/1661364508 >> 172.21.15.49:0/1661364508 conn(0x7f15d40f0120 msgr2=0x7f15d40f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:07.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.377+0000 7f15db034700 1 -- 172.21.15.49:0/1661364508 shutdown_connections 2023-11-30T21:49:07.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.377+0000 7f15db034700 1 -- 172.21.15.49:0/1661364508 wait complete. 2023-11-30T21:49:07.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.377+0000 7f15db034700 1 Processor -- start 2023-11-30T21:49:07.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.377+0000 7f15db034700 1 -- start start 2023-11-30T21:49:07.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.378+0000 7f15db034700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15d40f78e0 0x7f15d40f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:07.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.378+0000 7f15db034700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d40f5630 con 0x7f15d40f78e0 2023-11-30T21:49:07.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.378+0000 7f15da032700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15d40f78e0 0x7f15d40f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:07.380 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.378+0000 7f15da032700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15d40f78e0 0x7f15d40f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:41290/0 (socket says 172.21.15.49:41290) 2023-11-30T21:49:07.381 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.378+0000 7f15da032700 1 -- 172.21.15.49:0/3377360953 learned_addr learned my addr 172.21.15.49:0/3377360953 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:07.381 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.378+0000 7f15da032700 1 -- 172.21.15.49:0/3377360953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d0005040 con 0x7f15d40f78e0 2023-11-30T21:49:07.381 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.378+0000 7f15da032700 1 --2- 172.21.15.49:0/3377360953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15d40f78e0 0x7f15d40f71b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f15d0013510 tx=0x7f15d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:07.382 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.378+0000 7f15caffd700 1 -- 172.21.15.49:0/3377360953 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15d0009320 con 0x7f15d40f78e0 2023-11-30T21:49:07.382 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.378+0000 7f15caffd700 1 -- 172.21.15.49:0/3377360953 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15d0013f00 con 0x7f15d40f78e0 2023-11-30T21:49:07.382 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.378+0000 7f15db034700 1 -- 172.21.15.49:0/3377360953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d40f57d0 con 0x7f15d40f78e0 2023-11-30T21:49:07.382 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.378+0000 7f15caffd700 1 -- 172.21.15.49:0/3377360953 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15d0014dc0 con 0x7f15d40f78e0 2023-11-30T21:49:07.382 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.378+0000 7f15db034700 1 -- 172.21.15.49:0/3377360953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d40f5c60 con 0x7f15d40f78e0 2023-11-30T21:49:07.382 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.379+0000 7f15caffd700 1 -- 172.21.15.49:0/3377360953 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15d001d3d0 con 0x7f15d40f78e0 2023-11-30T21:49:07.382 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.379+0000 7f15caffd700 1 --2- 172.21.15.49:0/3377360953 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f15c002ff60 0x7f15c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:07.382 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.379+0000 7f15db034700 1 -- 172.21.15.49:0/3377360953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d4059100 con 0x7f15d40f78e0 2023-11-30T21:49:07.383 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.379+0000 7f15caffd700 1 -- 172.21.15.49:0/3377360953 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f15d0041050 con 0x7f15d40f78e0 2023-11-30T21:49:07.383 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.380+0000 7f15d9831700 1 --2- 172.21.15.49:0/3377360953 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f15c002ff60 0x7f15c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:07.383 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.380+0000 7f15d9831700 1 --2- 172.21.15.49:0/3377360953 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f15c002ff60 0x7f15c0032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f15c40096a0 tx=0x7f15c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:07.385 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.384+0000 7f15caffd700 1 -- 172.21.15.49:0/3377360953 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15d0012920 con 0x7f15d40f78e0 2023-11-30T21:49:07.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:07 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:05.505945+0000 mgr.smithi027.brrjcl (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:07.605 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.603+0000 7f15db034700 1 -- 172.21.15.49:0/3377360953 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15d40fbce0 con 0x7f15d40f78e0 2023-11-30T21:49:07.605 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.603+0000 7f15caffd700 1 -- 172.21.15.49:0/3377360953 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f15d0012330 con 0x7f15d40f78e0 2023-11-30T21:49:07.605 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:07.606 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:07.609 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.605+0000 7f15db034700 1 -- 172.21.15.49:0/3377360953 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f15c002ff60 msgr2=0x7f15c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:07.609 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.605+0000 7f15db034700 1 --2- 172.21.15.49:0/3377360953 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f15c002ff60 0x7f15c0032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f15c40096a0 tx=0x7f15c4006b40).stop 2023-11-30T21:49:07.609 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.605+0000 7f15db034700 1 -- 172.21.15.49:0/3377360953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15d40f78e0 msgr2=0x7f15d40f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:07.609 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.605+0000 7f15db034700 1 --2- 172.21.15.49:0/3377360953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15d40f78e0 0x7f15d40f71b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f15d0013510 tx=0x7f15d000b040).stop 2023-11-30T21:49:07.609 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.605+0000 7f15db034700 1 -- 172.21.15.49:0/3377360953 shutdown_connections 2023-11-30T21:49:07.609 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.605+0000 7f15db034700 1 --2- 172.21.15.49:0/3377360953 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f15c002ff60 0x7f15c0032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:07.609 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.605+0000 7f15db034700 1 --2- 172.21.15.49:0/3377360953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15d40f78e0 0x7f15d40f71b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:07.609 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.605+0000 7f15db034700 1 -- 172.21.15.49:0/3377360953 >> 172.21.15.49:0/3377360953 conn(0x7f15d40f0120 msgr2=0x7f15d40f0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:07.609 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.605+0000 7f15db034700 1 -- 172.21.15.49:0/3377360953 shutdown_connections 2023-11-30T21:49:07.609 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:07.605+0000 7f15db034700 1 -- 172.21.15.49:0/3377360953 wait complete. 2023-11-30T21:49:07.610 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:08.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:08 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:07.605029+0000 mon.smithi027 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.49:0/3377360953' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:09.048 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:09.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:09.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:09 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:07.506218+0000 mgr.smithi027.brrjcl (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:10.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.365+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3363316076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23b40f6510 msgr2=0x7f23b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:10.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.365+0000 7f23bc4b0700 1 --2- 172.21.15.49:0/3363316076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23b40f6510 0x7f23b40f6930 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f23b0009fb0 tx=0x7f23b0009b30).stop 2023-11-30T21:49:10.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.366+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3363316076 shutdown_connections 2023-11-30T21:49:10.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.366+0000 7f23bc4b0700 1 --2- 172.21.15.49:0/3363316076 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23b40f6510 0x7f23b40f6930 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:10.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.366+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3363316076 >> 172.21.15.49:0/3363316076 conn(0x7f23b40f1c70 msgr2=0x7f23b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:10.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.366+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3363316076 shutdown_connections 2023-11-30T21:49:10.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.366+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3363316076 wait complete. 2023-11-30T21:49:10.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.366+0000 7f23bc4b0700 1 Processor -- start 2023-11-30T21:49:10.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.366+0000 7f23bc4b0700 1 -- start start 2023-11-30T21:49:10.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.366+0000 7f23bc4b0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23b40f6510 0x7f23b4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:10.369 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.366+0000 7f23bc4b0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b4106c20 con 0x7f23b40f6510 2023-11-30T21:49:10.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.367+0000 7f23ba24c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23b40f6510 0x7f23b4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:10.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.367+0000 7f23ba24c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23b40f6510 0x7f23b4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:36828/0 (socket says 172.21.15.49:36828) 2023-11-30T21:49:10.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.367+0000 7f23ba24c700 1 -- 172.21.15.49:0/3489847151 learned_addr learned my addr 172.21.15.49:0/3489847151 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:10.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.367+0000 7f23ba24c700 1 -- 172.21.15.49:0/3489847151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b0005040 con 0x7f23b40f6510 2023-11-30T21:49:10.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.367+0000 7f23ba24c700 1 --2- 172.21.15.49:0/3489847151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23b40f6510 0x7f23b4106730 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f23b0013510 tx=0x7f23b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:10.371 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.367+0000 7f23a6ffd700 1 -- 172.21.15.49:0/3489847151 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23b0009320 con 0x7f23b40f6510 2023-11-30T21:49:10.371 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.367+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3489847151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b4106e20 con 0x7f23b40f6510 2023-11-30T21:49:10.371 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.367+0000 7f23a6ffd700 1 -- 172.21.15.49:0/3489847151 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23b0013f00 con 0x7f23b40f6510 2023-11-30T21:49:10.371 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.367+0000 7f23a6ffd700 1 -- 172.21.15.49:0/3489847151 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23b0014d90 con 0x7f23b40f6510 2023-11-30T21:49:10.371 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.367+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3489847151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b41072b0 con 0x7f23b40f6510 2023-11-30T21:49:10.372 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.368+0000 7f23a6ffd700 1 -- 172.21.15.49:0/3489847151 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23b00095f0 con 0x7f23b40f6510 2023-11-30T21:49:10.372 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.368+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3489847151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b4045ad0 con 0x7f23b40f6510 2023-11-30T21:49:10.372 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.368+0000 7f23a6ffd700 1 --2- 172.21.15.49:0/3489847151 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f23a002ff60 0x7f23a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:10.372 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.368+0000 7f23a6ffd700 1 -- 172.21.15.49:0/3489847151 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f23b00410a0 con 0x7f23b40f6510 2023-11-30T21:49:10.372 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.369+0000 7f23b9a4b700 1 --2- 172.21.15.49:0/3489847151 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f23a002ff60 0x7f23a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:10.372 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.369+0000 7f23b9a4b700 1 --2- 172.21.15.49:0/3489847151 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f23a002ff60 0x7f23a0032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f23a80096a0 tx=0x7f23a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:10.374 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.373+0000 7f23a6ffd700 1 -- 172.21.15.49:0/3489847151 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23b001d7f0 con 0x7f23b40f6510 2023-11-30T21:49:10.591 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.589+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3489847151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23b40fbf60 con 0x7f23b40f6510 2023-11-30T21:49:10.592 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.590+0000 7f23a6ffd700 1 -- 172.21.15.49:0/3489847151 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f23b0010230 con 0x7f23b40f6510 2023-11-30T21:49:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:10.592 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:10.594 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.591+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3489847151 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f23a002ff60 msgr2=0x7f23a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:10.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.591+0000 7f23bc4b0700 1 --2- 172.21.15.49:0/3489847151 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f23a002ff60 0x7f23a0032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f23a80096a0 tx=0x7f23a8006b40).stop 2023-11-30T21:49:10.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.591+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3489847151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23b40f6510 msgr2=0x7f23b4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:10.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.591+0000 7f23bc4b0700 1 --2- 172.21.15.49:0/3489847151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23b40f6510 0x7f23b4106730 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f23b0013510 tx=0x7f23b000b040).stop 2023-11-30T21:49:10.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.591+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3489847151 shutdown_connections 2023-11-30T21:49:10.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.591+0000 7f23bc4b0700 1 --2- 172.21.15.49:0/3489847151 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f23a002ff60 0x7f23a0032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:10.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.591+0000 7f23bc4b0700 1 --2- 172.21.15.49:0/3489847151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23b40f6510 0x7f23b4106730 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:10.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.591+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3489847151 >> 172.21.15.49:0/3489847151 conn(0x7f23b40f1c70 msgr2=0x7f23b40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:10.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.592+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3489847151 shutdown_connections 2023-11-30T21:49:10.595 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:10.592+0000 7f23bc4b0700 1 -- 172.21.15.49:0/3489847151 wait complete. 2023-11-30T21:49:10.595 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:11.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:09.506615+0000 mgr.smithi027.brrjcl (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:11.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:10.591320+0000 mon.smithi027 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.49:0/3489847151' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:12.015 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:12.015 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:13.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.251+0000 7ff99e47b700 1 -- 172.21.15.49:0/2451457111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9980f51a0 msgr2=0x7ff9980f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:13.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.251+0000 7ff99e47b700 1 --2- 172.21.15.49:0/2451457111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9980f51a0 0x7ff9980f55c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7ff988009fb0 tx=0x7ff988009b30).stop 2023-11-30T21:49:13.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.251+0000 7ff99e47b700 1 -- 172.21.15.49:0/2451457111 shutdown_connections 2023-11-30T21:49:13.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.251+0000 7ff99e47b700 1 --2- 172.21.15.49:0/2451457111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9980f51a0 0x7ff9980f55c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:13.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.251+0000 7ff99e47b700 1 -- 172.21.15.49:0/2451457111 >> 172.21.15.49:0/2451457111 conn(0x7ff9980f09a0 msgr2=0x7ff9980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:13.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.251+0000 7ff99e47b700 1 -- 172.21.15.49:0/2451457111 shutdown_connections 2023-11-30T21:49:13.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.251+0000 7ff99e47b700 1 -- 172.21.15.49:0/2451457111 wait complete. 2023-11-30T21:49:13.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.252+0000 7ff99e47b700 1 Processor -- start 2023-11-30T21:49:13.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.252+0000 7ff99e47b700 1 -- start start 2023-11-30T21:49:13.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.252+0000 7ff99e47b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9980f51a0 0x7ff9981091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:13.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.252+0000 7ff99e47b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff998060220 con 0x7ff9980f51a0 2023-11-30T21:49:13.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.252+0000 7ff99d479700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9980f51a0 0x7ff9981091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:13.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.252+0000 7ff99d479700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9980f51a0 0x7ff9981091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:36848/0 (socket says 172.21.15.49:36848) 2023-11-30T21:49:13.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.252+0000 7ff99d479700 1 -- 172.21.15.49:0/1344294328 learned_addr learned my addr 172.21.15.49:0/1344294328 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:13.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.253+0000 7ff99d479700 1 -- 172.21.15.49:0/1344294328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff988005040 con 0x7ff9980f51a0 2023-11-30T21:49:13.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.253+0000 7ff99d479700 1 --2- 172.21.15.49:0/1344294328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9980f51a0 0x7ff9981091d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7ff988009830 tx=0x7ff98800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:13.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.253+0000 7ff98dffb700 1 -- 172.21.15.49:0/1344294328 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff988009320 con 0x7ff9980f51a0 2023-11-30T21:49:13.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.253+0000 7ff98dffb700 1 -- 172.21.15.49:0/1344294328 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9880136c0 con 0x7ff9980f51a0 2023-11-30T21:49:13.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.253+0000 7ff99e47b700 1 -- 172.21.15.49:0/1344294328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff998060420 con 0x7ff9980f51a0 2023-11-30T21:49:13.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.253+0000 7ff98dffb700 1 -- 172.21.15.49:0/1344294328 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff98801ee90 con 0x7ff9980f51a0 2023-11-30T21:49:13.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.253+0000 7ff99e47b700 1 -- 172.21.15.49:0/1344294328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff998109880 con 0x7ff9980f51a0 2023-11-30T21:49:13.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.254+0000 7ff98dffb700 1 -- 172.21.15.49:0/1344294328 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff98801e420 con 0x7ff9980f51a0 2023-11-30T21:49:13.257 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.254+0000 7ff99e47b700 1 -- 172.21.15.49:0/1344294328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff998059100 con 0x7ff9980f51a0 2023-11-30T21:49:13.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.254+0000 7ff98dffb700 1 --2- 172.21.15.49:0/1344294328 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff98402ff10 0x7ff9840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:13.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.254+0000 7ff98dffb700 1 -- 172.21.15.49:0/1344294328 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9880410b0 con 0x7ff9980f51a0 2023-11-30T21:49:13.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.255+0000 7ff99cc78700 1 --2- 172.21.15.49:0/1344294328 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff98402ff10 0x7ff9840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:13.258 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.255+0000 7ff99cc78700 1 --2- 172.21.15.49:0/1344294328 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff98402ff10 0x7ff9840323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7ff9940096a0 tx=0x7ff994006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:13.260 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.259+0000 7ff98dffb700 1 -- 172.21.15.49:0/1344294328 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9880095f0 con 0x7ff9980f51a0 2023-11-30T21:49:13.482 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.480+0000 7ff99e47b700 1 -- 172.21.15.49:0/1344294328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9980fabe0 con 0x7ff9980f51a0 2023-11-30T21:49:13.482 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.481+0000 7ff98dffb700 1 -- 172.21.15.49:0/1344294328 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff98802c2d0 con 0x7ff9980f51a0 2023-11-30T21:49:13.483 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:13.483 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:13.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.482+0000 7ff99e47b700 1 -- 172.21.15.49:0/1344294328 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff98402ff10 msgr2=0x7ff9840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:13.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.482+0000 7ff99e47b700 1 --2- 172.21.15.49:0/1344294328 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff98402ff10 0x7ff9840323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7ff9940096a0 tx=0x7ff994006b40).stop 2023-11-30T21:49:13.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.482+0000 7ff99e47b700 1 -- 172.21.15.49:0/1344294328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9980f51a0 msgr2=0x7ff9981091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:13.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.482+0000 7ff99e47b700 1 --2- 172.21.15.49:0/1344294328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9980f51a0 0x7ff9981091d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7ff988009830 tx=0x7ff98800b040).stop 2023-11-30T21:49:13.486 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.483+0000 7ff99e47b700 1 -- 172.21.15.49:0/1344294328 shutdown_connections 2023-11-30T21:49:13.486 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.483+0000 7ff99e47b700 1 --2- 172.21.15.49:0/1344294328 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff98402ff10 0x7ff9840323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:13.486 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.483+0000 7ff99e47b700 1 --2- 172.21.15.49:0/1344294328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9980f51a0 0x7ff9981091d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:13.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.483+0000 7ff99e47b700 1 -- 172.21.15.49:0/1344294328 >> 172.21.15.49:0/1344294328 conn(0x7ff9980f09a0 msgr2=0x7ff9980f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:13.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.483+0000 7ff99e47b700 1 -- 172.21.15.49:0/1344294328 shutdown_connections 2023-11-30T21:49:13.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:13.483+0000 7ff99e47b700 1 -- 172.21.15.49:0/1344294328 wait complete. 2023-11-30T21:49:13.487 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:13.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:13 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:11.507091+0000 mgr.smithi027.brrjcl (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:14.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:13.482359+0000 mon.smithi027 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.49:0/1344294328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:14.893 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:14.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:15.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:15 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:13.507305+0000 mgr.smithi027.brrjcl (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:16.123 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.117+0000 7f83ad069700 1 -- 172.21.15.49:0/3675253924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a80f62a0 msgr2=0x7f83a80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:16.123 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.117+0000 7f83ad069700 1 --2- 172.21.15.49:0/3675253924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a80f62a0 0x7f83a80f66c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f8398009fb0 tx=0x7f8398009b30).stop 2023-11-30T21:49:16.123 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.117+0000 7f83ad069700 1 -- 172.21.15.49:0/3675253924 shutdown_connections 2023-11-30T21:49:16.123 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.117+0000 7f83ad069700 1 --2- 172.21.15.49:0/3675253924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a80f62a0 0x7f83a80f66c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:16.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.117+0000 7f83ad069700 1 -- 172.21.15.49:0/3675253924 >> 172.21.15.49:0/3675253924 conn(0x7f83a80f1aa0 msgr2=0x7f83a80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:16.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.117+0000 7f83ad069700 1 -- 172.21.15.49:0/3675253924 shutdown_connections 2023-11-30T21:49:16.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.117+0000 7f83ad069700 1 -- 172.21.15.49:0/3675253924 wait complete. 2023-11-30T21:49:16.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.118+0000 7f83ad069700 1 Processor -- start 2023-11-30T21:49:16.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.118+0000 7f83ad069700 1 -- start start 2023-11-30T21:49:16.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.119+0000 7f83ad069700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a80f62a0 0x7f83a81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:16.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.119+0000 7f83ad069700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a81069a0 con 0x7f83a80f62a0 2023-11-30T21:49:16.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.119+0000 7f83a7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a80f62a0 0x7f83a81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:16.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.119+0000 7f83a7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a80f62a0 0x7f83a81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:36862/0 (socket says 172.21.15.49:36862) 2023-11-30T21:49:16.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.119+0000 7f83a7fff700 1 -- 172.21.15.49:0/759003909 learned_addr learned my addr 172.21.15.49:0/759003909 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:16.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.119+0000 7f83a7fff700 1 -- 172.21.15.49:0/759003909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8398005040 con 0x7f83a80f62a0 2023-11-30T21:49:16.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.119+0000 7f83a7fff700 1 --2- 172.21.15.49:0/759003909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a80f62a0 0x7f83a81064b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f8398013510 tx=0x7f839800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:16.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.119+0000 7f83a4ff9700 1 -- 172.21.15.49:0/759003909 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8398009320 con 0x7f83a80f62a0 2023-11-30T21:49:16.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.119+0000 7f83a4ff9700 1 -- 172.21.15.49:0/759003909 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8398013f00 con 0x7f83a80f62a0 2023-11-30T21:49:16.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.119+0000 7f83ad069700 1 -- 172.21.15.49:0/759003909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a8106ba0 con 0x7f83a80f62a0 2023-11-30T21:49:16.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.119+0000 7f83a4ff9700 1 -- 172.21.15.49:0/759003909 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8398014dc0 con 0x7f83a80f62a0 2023-11-30T21:49:16.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.119+0000 7f83ad069700 1 -- 172.21.15.49:0/759003909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a8107030 con 0x7f83a80f62a0 2023-11-30T21:49:16.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.120+0000 7f83a4ff9700 1 -- 172.21.15.49:0/759003909 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83980263a0 con 0x7f83a80f62a0 2023-11-30T21:49:16.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.120+0000 7f83a4ff9700 1 --2- 172.21.15.49:0/759003909 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f839002ffb0 0x7f8390032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:16.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.120+0000 7f83a4ff9700 1 -- 172.21.15.49:0/759003909 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83980410b0 con 0x7f83a80f62a0 2023-11-30T21:49:16.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.120+0000 7f83ad069700 1 -- 172.21.15.49:0/759003909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a8045ad0 con 0x7f83a80f62a0 2023-11-30T21:49:16.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.120+0000 7f83a77fe700 1 --2- 172.21.15.49:0/759003909 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f839002ffb0 0x7f8390032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:16.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.121+0000 7f83a77fe700 1 --2- 172.21.15.49:0/759003909 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f839002ffb0 0x7f8390032470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f839c0096a0 tx=0x7f839c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:16.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.125+0000 7f83a4ff9700 1 -- 172.21.15.49:0/759003909 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8398012920 con 0x7f83a80f62a0 2023-11-30T21:49:16.194 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:16 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:15.534460+0000 mon.smithi027 (mon.0) 221 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-30T21:49:16.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:16 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:15.537119+0000 mon.smithi027 (mon.0) 222 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-30T21:49:16.195 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:16 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:15.537952+0000 mon.smithi027 (mon.0) 223 : audit [DBG] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-30T21:49:16.350 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.346+0000 7f83ad069700 1 -- 172.21.15.49:0/759003909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83a8107d50 con 0x7f83a80f62a0 2023-11-30T21:49:16.350 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.347+0000 7f83a4ff9700 1 -- 172.21.15.49:0/759003909 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f839802b020 con 0x7f83a80f62a0 2023-11-30T21:49:16.350 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:16.351 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:16.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.349+0000 7f83ad069700 1 -- 172.21.15.49:0/759003909 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f839002ffb0 msgr2=0x7f8390032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:16.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.349+0000 7f83ad069700 1 --2- 172.21.15.49:0/759003909 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f839002ffb0 0x7f8390032470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f839c0096a0 tx=0x7f839c006b40).stop 2023-11-30T21:49:16.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.349+0000 7f83ad069700 1 -- 172.21.15.49:0/759003909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a80f62a0 msgr2=0x7f83a81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:16.353 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.349+0000 7f83ad069700 1 --2- 172.21.15.49:0/759003909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a80f62a0 0x7f83a81064b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f8398013510 tx=0x7f839800b040).stop 2023-11-30T21:49:16.353 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.349+0000 7f83ad069700 1 -- 172.21.15.49:0/759003909 shutdown_connections 2023-11-30T21:49:16.353 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.349+0000 7f83ad069700 1 --2- 172.21.15.49:0/759003909 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f839002ffb0 0x7f8390032470 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:16.353 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.349+0000 7f83ad069700 1 --2- 172.21.15.49:0/759003909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83a80f62a0 0x7f83a81064b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:16.353 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.349+0000 7f83ad069700 1 -- 172.21.15.49:0/759003909 >> 172.21.15.49:0/759003909 conn(0x7f83a80f1aa0 msgr2=0x7f83a80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:16.353 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.349+0000 7f83ad069700 1 -- 172.21.15.49:0/759003909 shutdown_connections 2023-11-30T21:49:16.354 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:16.349+0000 7f83ad069700 1 -- 172.21.15.49:0/759003909 wait complete. 2023-11-30T21:49:16.354 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:17.261 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:15.507564+0000 mgr.smithi027.brrjcl (mgr.14162) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:17.261 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:49:15.533234+0000 mgr.smithi027.brrjcl (mgr.14162) 158 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi027 ... 2023-11-30T21:49:17.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-30T21:49:17.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-30T21:49:17.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr time="2023-11-30T21:44:54Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-30T21:49:17.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr time="2023-11-30T21:45:31Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-30T21:49:17.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr time="2023-11-30T21:46:09Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-30T21:49:17.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-30T21:49:17.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-30T21:49:17.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-30T21:49:17.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr time="2023-11-30T21:47:23Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-30T21:49:17.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr time="2023-11-30T21:48:00Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-30T21:49:17.262 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr time="2023-11-30T21:48:37Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-30T21:49:17.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-30T21:49:17.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Traceback (most recent call last): 2023-11-30T21:49:17.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-30T21:49:17.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: main() 2023-11-30T21:49:17.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-30T21:49:17.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: r = ctx.func(ctx) 2023-11-30T21:49:17.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-30T21:49:17.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: return func(ctx) 2023-11-30T21:49:17.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-30T21:49:17.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-30T21:49:17.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-30T21:49:17.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-30T21:49:17.264 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-30T21:49:17.264 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: raise RuntimeError('uid/gid not found') 2023-11-30T21:49:17.264 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: RuntimeError: uid/gid not found 2023-11-30T21:49:17.264 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Traceback (most recent call last): 2023-11-30T21:49:17.264 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-30T21:49:17.264 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: yield (conn, connr) 2023-11-30T21:49:17.264 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-30T21:49:17.264 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: code, '\n'.join(err))) 2023-11-30T21:49:17.264 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi027 ... 2023-11-30T21:49:17.264 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-30T21:49:17.264 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-30T21:49:17.264 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr time="2023-11-30T21:44:54Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-30T21:49:17.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr time="2023-11-30T21:45:31Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-30T21:49:17.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr time="2023-11-30T21:46:09Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-30T21:49:17.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-30T21:49:17.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi027 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-30T21:49:17.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-30T21:49:17.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr time="2023-11-30T21:47:23Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-30T21:49:17.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr time="2023-11-30T21:48:00Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-30T21:49:17.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr time="2023-11-30T21:48:37Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-30T21:49:17.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-30T21:49:17.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Traceback (most recent call last): 2023-11-30T21:49:17.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-30T21:49:17.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: main() 2023-11-30T21:49:17.266 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-30T21:49:17.266 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: r = ctx.func(ctx) 2023-11-30T21:49:17.266 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-30T21:49:17.266 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: return func(ctx) 2023-11-30T21:49:17.266 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-30T21:49:17.266 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-30T21:49:17.266 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-30T21:49:17.266 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-30T21:49:17.266 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-30T21:49:17.266 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: raise RuntimeError('uid/gid not found') 2023-11-30T21:49:17.266 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: RuntimeError: uid/gid not found 2023-11-30T21:49:17.266 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:49:15.538673+0000 mgr.smithi027.brrjcl (mgr.14162) 159 : cephadm [INF] Deploying daemon crash.smithi027 on smithi027 2023-11-30T21:49:17.267 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:16.348774+0000 mon.smithi027 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.49:0/759003909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:17.267 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:17.011896+0000 mon.smithi027 (mon.0) 225 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:49:17.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:17.763 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:18.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:18 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:17.227745+0000 mon.smithi027 (mon.0) 226 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:49:18.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:18 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:17.229757+0000 mon.smithi027 (mon.0) 227 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:49:18.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:18 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:17.230179+0000 mon.smithi027 (mon.0) 228 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-30T21:49:18.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:18 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:17.230592+0000 mgr.smithi027.brrjcl (mgr.14162) 160 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-30T21:49:18.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:18 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:17.232883+0000 mon.smithi027 (mon.0) 229 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:49:18.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:18 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:49:17.236916+0000 mgr.smithi027.brrjcl (mgr.14162) 161 : cephadm [INF] Deploying daemon grafana.smithi027 on smithi027 2023-11-30T21:49:18.978 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.977+0000 7fcfea839700 1 -- 172.21.15.49:0/1946840182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe4060220 msgr2=0x7fcfe40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:18.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.977+0000 7fcfea839700 1 --2- 172.21.15.49:0/1946840182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe4060220 0x7fcfe40f9150 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fcfd4009fb0 tx=0x7fcfd4009b30).stop 2023-11-30T21:49:18.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.977+0000 7fcfea839700 1 -- 172.21.15.49:0/1946840182 shutdown_connections 2023-11-30T21:49:18.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.977+0000 7fcfea839700 1 --2- 172.21.15.49:0/1946840182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe4060220 0x7fcfe40f9150 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:18.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.977+0000 7fcfea839700 1 -- 172.21.15.49:0/1946840182 >> 172.21.15.49:0/1946840182 conn(0x7fcfe40f0b70 msgr2=0x7fcfe40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:18.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.977+0000 7fcfea839700 1 -- 172.21.15.49:0/1946840182 shutdown_connections 2023-11-30T21:49:18.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.977+0000 7fcfea839700 1 -- 172.21.15.49:0/1946840182 wait complete. 2023-11-30T21:49:18.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.977+0000 7fcfea839700 1 Processor -- start 2023-11-30T21:49:18.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.978+0000 7fcfea839700 1 -- start start 2023-11-30T21:49:18.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.978+0000 7fcfea839700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe4060220 0x7fcfe40f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:18.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.978+0000 7fcfea839700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40f5050 con 0x7fcfe4060220 2023-11-30T21:49:18.980 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.978+0000 7fcfe3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe4060220 0x7fcfe40f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:18.980 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.978+0000 7fcfe3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe4060220 0x7fcfe40f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:48868/0 (socket says 172.21.15.49:48868) 2023-11-30T21:49:18.980 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.978+0000 7fcfe3fff700 1 -- 172.21.15.49:0/963608543 learned_addr learned my addr 172.21.15.49:0/963608543 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:18.981 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.979+0000 7fcfe3fff700 1 -- 172.21.15.49:0/963608543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd4005040 con 0x7fcfe4060220 2023-11-30T21:49:18.981 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.979+0000 7fcfe3fff700 1 --2- 172.21.15.49:0/963608543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe4060220 0x7fcfe40f6bd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fcfd4013510 tx=0x7fcfd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:18.981 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.979+0000 7fcfe0ff9700 1 -- 172.21.15.49:0/963608543 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfd4009320 con 0x7fcfe4060220 2023-11-30T21:49:18.981 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.979+0000 7fcfea839700 1 -- 172.21.15.49:0/963608543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe40f51f0 con 0x7fcfe4060220 2023-11-30T21:49:18.981 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.979+0000 7fcfe0ff9700 1 -- 172.21.15.49:0/963608543 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfd4013f00 con 0x7fcfe4060220 2023-11-30T21:49:18.981 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.979+0000 7fcfe0ff9700 1 -- 172.21.15.49:0/963608543 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfd4014dc0 con 0x7fcfe4060220 2023-11-30T21:49:18.981 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.979+0000 7fcfea839700 1 -- 172.21.15.49:0/963608543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe40f5680 con 0x7fcfe4060220 2023-11-30T21:49:18.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.980+0000 7fcfe0ff9700 1 -- 172.21.15.49:0/963608543 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfd401d3d0 con 0x7fcfe4060220 2023-11-30T21:49:18.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.980+0000 7fcfea839700 1 -- 172.21.15.49:0/963608543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe4059100 con 0x7fcfe4060220 2023-11-30T21:49:18.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.980+0000 7fcfe0ff9700 1 --2- 172.21.15.49:0/963608543 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcfcc02ff60 0x7fcfcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:18.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.980+0000 7fcfe0ff9700 1 -- 172.21.15.49:0/963608543 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfd4041320 con 0x7fcfe4060220 2023-11-30T21:49:18.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.981+0000 7fcfe37fe700 1 --2- 172.21.15.49:0/963608543 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcfcc02ff60 0x7fcfcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:18.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.981+0000 7fcfe37fe700 1 --2- 172.21.15.49:0/963608543 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcfcc02ff60 0x7fcfcc032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fcfd80096a0 tx=0x7fcfd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:18.985 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:18.985+0000 7fcfe0ff9700 1 -- 172.21.15.49:0/963608543 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfd401cb50 con 0x7fcfe4060220 2023-11-30T21:49:19.203 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:19.203+0000 7fcfea839700 1 -- 172.21.15.49:0/963608543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfe40fbf50 con 0x7fcfe4060220 2023-11-30T21:49:19.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:19.203+0000 7fcfe0ff9700 1 -- 172.21.15.49:0/963608543 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcfd401c970 con 0x7fcfe4060220 2023-11-30T21:49:19.204 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:19.204 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:19.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:19.205+0000 7fcfea839700 1 -- 172.21.15.49:0/963608543 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcfcc02ff60 msgr2=0x7fcfcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:19.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:19.205+0000 7fcfea839700 1 --2- 172.21.15.49:0/963608543 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcfcc02ff60 0x7fcfcc032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fcfd80096a0 tx=0x7fcfd8006b40).stop 2023-11-30T21:49:19.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:19.205+0000 7fcfea839700 1 -- 172.21.15.49:0/963608543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe4060220 msgr2=0x7fcfe40f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:19.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:19.205+0000 7fcfea839700 1 --2- 172.21.15.49:0/963608543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe4060220 0x7fcfe40f6bd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fcfd4013510 tx=0x7fcfd400b040).stop 2023-11-30T21:49:19.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:19.205+0000 7fcfea839700 1 -- 172.21.15.49:0/963608543 shutdown_connections 2023-11-30T21:49:19.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:19.205+0000 7fcfea839700 1 --2- 172.21.15.49:0/963608543 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fcfcc02ff60 0x7fcfcc032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:19.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:19.205+0000 7fcfea839700 1 --2- 172.21.15.49:0/963608543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcfe4060220 0x7fcfe40f6bd0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:19.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:19.205+0000 7fcfea839700 1 -- 172.21.15.49:0/963608543 >> 172.21.15.49:0/963608543 conn(0x7fcfe40f0b70 msgr2=0x7fcfe40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:19.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:19.205+0000 7fcfea839700 1 -- 172.21.15.49:0/963608543 shutdown_connections 2023-11-30T21:49:19.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:19.205+0000 7fcfea839700 1 -- 172.21.15.49:0/963608543 wait complete. 2023-11-30T21:49:19.207 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:19.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:17.508050+0000 mgr.smithi027.brrjcl (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:19.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:19.203968+0000 mon.smithi027 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.49:0/963608543' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:20.634 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:20.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:21.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:21 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:19.508451+0000 mgr.smithi027.brrjcl (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:21.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.865+0000 7f38bb95b700 1 -- 172.21.15.49:0/3250268134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b40f5410 msgr2=0x7f38b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:21.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.865+0000 7f38bb95b700 1 --2- 172.21.15.49:0/3250268134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b40f5410 0x7f38b40f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f38b0009fb0 tx=0x7f38b0009b30).stop 2023-11-30T21:49:21.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.865+0000 7f38bb95b700 1 -- 172.21.15.49:0/3250268134 shutdown_connections 2023-11-30T21:49:21.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.865+0000 7f38bb95b700 1 --2- 172.21.15.49:0/3250268134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b40f5410 0x7f38b40f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:21.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.865+0000 7f38bb95b700 1 -- 172.21.15.49:0/3250268134 >> 172.21.15.49:0/3250268134 conn(0x7f38b40f0b70 msgr2=0x7f38b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:21.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.865+0000 7f38bb95b700 1 -- 172.21.15.49:0/3250268134 shutdown_connections 2023-11-30T21:49:21.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.865+0000 7f38bb95b700 1 -- 172.21.15.49:0/3250268134 wait complete. 2023-11-30T21:49:21.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.866+0000 7f38bb95b700 1 Processor -- start 2023-11-30T21:49:21.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.866+0000 7f38bb95b700 1 -- start start 2023-11-30T21:49:21.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.866+0000 7f38bb95b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b40f5410 0x7f38b4106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:21.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.866+0000 7f38bb95b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b4106c00 con 0x7f38b40f5410 2023-11-30T21:49:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.866+0000 7f38b96f7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b40f5410 0x7f38b4106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.867+0000 7f38b96f7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b40f5410 0x7f38b4106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:48894/0 (socket says 172.21.15.49:48894) 2023-11-30T21:49:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.867+0000 7f38b96f7700 1 -- 172.21.15.49:0/2791000516 learned_addr learned my addr 172.21.15.49:0/2791000516 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.867+0000 7f38b96f7700 1 -- 172.21.15.49:0/2791000516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b0005040 con 0x7f38b40f5410 2023-11-30T21:49:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.867+0000 7f38b96f7700 1 --2- 172.21.15.49:0/2791000516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b40f5410 0x7f38b4106710 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f38b0013510 tx=0x7f38b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:21.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.867+0000 7f38a9ffb700 1 -- 172.21.15.49:0/2791000516 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38b0009320 con 0x7f38b40f5410 2023-11-30T21:49:21.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.867+0000 7f38bb95b700 1 -- 172.21.15.49:0/2791000516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b4106e00 con 0x7f38b40f5410 2023-11-30T21:49:21.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.867+0000 7f38a9ffb700 1 -- 172.21.15.49:0/2791000516 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38b0013f00 con 0x7f38b40f5410 2023-11-30T21:49:21.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.867+0000 7f38a9ffb700 1 -- 172.21.15.49:0/2791000516 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38b0014d90 con 0x7f38b40f5410 2023-11-30T21:49:21.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.867+0000 7f38bb95b700 1 -- 172.21.15.49:0/2791000516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b4107290 con 0x7f38b40f5410 2023-11-30T21:49:21.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.868+0000 7f38a9ffb700 1 -- 172.21.15.49:0/2791000516 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38b00095f0 con 0x7f38b40f5410 2023-11-30T21:49:21.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.868+0000 7f38bb95b700 1 -- 172.21.15.49:0/2791000516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b4045ad0 con 0x7f38b40f5410 2023-11-30T21:49:21.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.868+0000 7f38a9ffb700 1 --2- 172.21.15.49:0/2791000516 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38a002ff60 0x7f38a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:21.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.868+0000 7f38a9ffb700 1 -- 172.21.15.49:0/2791000516 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38b00410c0 con 0x7f38b40f5410 2023-11-30T21:49:21.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.869+0000 7f38b8ef6700 1 --2- 172.21.15.49:0/2791000516 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38a002ff60 0x7f38a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:21.871 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.869+0000 7f38b8ef6700 1 --2- 172.21.15.49:0/2791000516 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38a002ff60 0x7f38a0032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f38a40096a0 tx=0x7f38a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:21.873 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:21.873+0000 7f38a9ffb700 1 -- 172.21.15.49:0/2791000516 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38b0014420 con 0x7f38b40f5410 2023-11-30T21:49:22.089 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:22.088+0000 7f38bb95b700 1 -- 172.21.15.49:0/2791000516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38b40fae60 con 0x7f38b40f5410 2023-11-30T21:49:22.089 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:22.089+0000 7f38a9ffb700 1 -- 172.21.15.49:0/2791000516 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f38b001d7f0 con 0x7f38b40f5410 2023-11-30T21:49:22.090 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:22.090 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:22.090+0000 7f38bb95b700 1 -- 172.21.15.49:0/2791000516 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38a002ff60 msgr2=0x7f38a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:22.090+0000 7f38bb95b700 1 --2- 172.21.15.49:0/2791000516 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38a002ff60 0x7f38a0032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f38a40096a0 tx=0x7f38a4006b40).stop 2023-11-30T21:49:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:22.090+0000 7f38bb95b700 1 -- 172.21.15.49:0/2791000516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b40f5410 msgr2=0x7f38b4106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:22.090+0000 7f38bb95b700 1 --2- 172.21.15.49:0/2791000516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b40f5410 0x7f38b4106710 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f38b0013510 tx=0x7f38b000b040).stop 2023-11-30T21:49:22.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:22.090+0000 7f38bb95b700 1 -- 172.21.15.49:0/2791000516 shutdown_connections 2023-11-30T21:49:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:22.090+0000 7f38bb95b700 1 --2- 172.21.15.49:0/2791000516 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f38a002ff60 0x7f38a0032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:22.090+0000 7f38bb95b700 1 --2- 172.21.15.49:0/2791000516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38b40f5410 0x7f38b4106710 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:22.090+0000 7f38bb95b700 1 -- 172.21.15.49:0/2791000516 >> 172.21.15.49:0/2791000516 conn(0x7f38b40f0b70 msgr2=0x7f38b405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:22.090+0000 7f38bb95b700 1 -- 172.21.15.49:0/2791000516 shutdown_connections 2023-11-30T21:49:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:22.090+0000 7f38bb95b700 1 -- 172.21.15.49:0/2791000516 wait complete. 2023-11-30T21:49:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:22.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:22 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:22.089326+0000 mon.smithi027 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.49:0/2791000516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:23.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:23 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:21.508791+0000 mgr.smithi027.brrjcl (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:23.520 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:23.520 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:24.756 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.756+0000 7f642df84700 1 -- 172.21.15.49:0/3749879005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6428060220 msgr2=0x7f64280f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:24.756 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.756+0000 7f642df84700 1 --2- 172.21.15.49:0/3749879005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6428060220 0x7f64280f9150 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f6418009fb0 tx=0x7f6418009b30).stop 2023-11-30T21:49:24.757 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.756+0000 7f642df84700 1 -- 172.21.15.49:0/3749879005 shutdown_connections 2023-11-30T21:49:24.757 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.756+0000 7f642df84700 1 --2- 172.21.15.49:0/3749879005 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6428060220 0x7f64280f9150 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:24.757 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.756+0000 7f642df84700 1 -- 172.21.15.49:0/3749879005 >> 172.21.15.49:0/3749879005 conn(0x7f64280f0b70 msgr2=0x7f64280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:24.757 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.756+0000 7f642df84700 1 -- 172.21.15.49:0/3749879005 shutdown_connections 2023-11-30T21:49:24.757 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.756+0000 7f642df84700 1 -- 172.21.15.49:0/3749879005 wait complete. 2023-11-30T21:49:24.758 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.756+0000 7f642df84700 1 Processor -- start 2023-11-30T21:49:24.758 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.756+0000 7f642df84700 1 -- start start 2023-11-30T21:49:24.758 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.757+0000 7f642df84700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6428060220 0x7f64281066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:24.758 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.757+0000 7f642df84700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6428106bd0 con 0x7f6428060220 2023-11-30T21:49:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.757+0000 7f64277fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6428060220 0x7f64281066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.757+0000 7f64277fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6428060220 0x7f64281066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:48904/0 (socket says 172.21.15.49:48904) 2023-11-30T21:49:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.757+0000 7f64277fe700 1 -- 172.21.15.49:0/926286545 learned_addr learned my addr 172.21.15.49:0/926286545 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.757+0000 7f64277fe700 1 -- 172.21.15.49:0/926286545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6418005040 con 0x7f6428060220 2023-11-30T21:49:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.757+0000 7f64277fe700 1 --2- 172.21.15.49:0/926286545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6428060220 0x7f64281066e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f6418013510 tx=0x7f641800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.757+0000 7f6424ff9700 1 -- 172.21.15.49:0/926286545 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6418009320 con 0x7f6428060220 2023-11-30T21:49:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.757+0000 7f642df84700 1 -- 172.21.15.49:0/926286545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6428106dd0 con 0x7f6428060220 2023-11-30T21:49:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.757+0000 7f6424ff9700 1 -- 172.21.15.49:0/926286545 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6418013f00 con 0x7f6428060220 2023-11-30T21:49:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.757+0000 7f6424ff9700 1 -- 172.21.15.49:0/926286545 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6418014dc0 con 0x7f6428060220 2023-11-30T21:49:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.757+0000 7f642df84700 1 -- 172.21.15.49:0/926286545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6428107260 con 0x7f6428060220 2023-11-30T21:49:24.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.758+0000 7f6424ff9700 1 -- 172.21.15.49:0/926286545 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6418014420 con 0x7f6428060220 2023-11-30T21:49:24.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.758+0000 7f642df84700 1 -- 172.21.15.49:0/926286545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6428045ad0 con 0x7f6428060220 2023-11-30T21:49:24.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.758+0000 7f6424ff9700 1 --2- 172.21.15.49:0/926286545 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f641402ff60 0x7f6414032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:24.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.758+0000 7f6424ff9700 1 -- 172.21.15.49:0/926286545 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6418041110 con 0x7f6428060220 2023-11-30T21:49:24.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.759+0000 7f641ffff700 1 --2- 172.21.15.49:0/926286545 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f641402ff60 0x7f6414032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:24.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.759+0000 7f641ffff700 1 --2- 172.21.15.49:0/926286545 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f641402ff60 0x7f6414032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f64100096a0 tx=0x7f6410006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:24.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.763+0000 7f6424ff9700 1 -- 172.21.15.49:0/926286545 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f641801cb10 con 0x7f6428060220 2023-11-30T21:49:24.979 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.978+0000 7f642df84700 1 -- 172.21.15.49:0/926286545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6428108110 con 0x7f6428060220 2023-11-30T21:49:24.980 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.979+0000 7f6424ff9700 1 -- 172.21.15.49:0/926286545 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f641801cb10 con 0x7f6428060220 2023-11-30T21:49:24.980 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:24.980 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:24.981 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.981+0000 7f642df84700 1 -- 172.21.15.49:0/926286545 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f641402ff60 msgr2=0x7f6414032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:24.981 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.981+0000 7f642df84700 1 --2- 172.21.15.49:0/926286545 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f641402ff60 0x7f6414032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f64100096a0 tx=0x7f6410006b40).stop 2023-11-30T21:49:24.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.981+0000 7f642df84700 1 -- 172.21.15.49:0/926286545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6428060220 msgr2=0x7f64281066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:24.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.981+0000 7f642df84700 1 --2- 172.21.15.49:0/926286545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6428060220 0x7f64281066e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f6418013510 tx=0x7f641800b040).stop 2023-11-30T21:49:24.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.981+0000 7f642df84700 1 -- 172.21.15.49:0/926286545 shutdown_connections 2023-11-30T21:49:24.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.981+0000 7f642df84700 1 --2- 172.21.15.49:0/926286545 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f641402ff60 0x7f6414032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:24.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.981+0000 7f642df84700 1 --2- 172.21.15.49:0/926286545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6428060220 0x7f64281066e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:24.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.981+0000 7f642df84700 1 -- 172.21.15.49:0/926286545 >> 172.21.15.49:0/926286545 conn(0x7f64280f0b70 msgr2=0x7f64280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:24.982 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.981+0000 7f642df84700 1 -- 172.21.15.49:0/926286545 shutdown_connections 2023-11-30T21:49:24.983 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:24.981+0000 7f642df84700 1 -- 172.21.15.49:0/926286545 wait complete. 2023-11-30T21:49:24.983 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:25.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:23.509215+0000 mgr.smithi027.brrjcl (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:25.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:24.979564+0000 mon.smithi027 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.49:0/926286545' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:26.393 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:26.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:27.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:27 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:25.509517+0000 mgr.smithi027.brrjcl (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:27.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.634+0000 7fdff431e700 1 -- 172.21.15.49:0/679034575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec0f6510 msgr2=0x7fdfec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:27.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.634+0000 7fdff431e700 1 --2- 172.21.15.49:0/679034575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec0f6510 0x7fdfec0f6930 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fdfdc009960 tx=0x7fdfdc004d10).stop 2023-11-30T21:49:27.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.634+0000 7fdff431e700 1 -- 172.21.15.49:0/679034575 shutdown_connections 2023-11-30T21:49:27.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.634+0000 7fdff431e700 1 --2- 172.21.15.49:0/679034575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec0f6510 0x7fdfec0f6930 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:27.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.634+0000 7fdff431e700 1 -- 172.21.15.49:0/679034575 >> 172.21.15.49:0/679034575 conn(0x7fdfec0f1c70 msgr2=0x7fdfec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:27.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.634+0000 7fdff431e700 1 -- 172.21.15.49:0/679034575 shutdown_connections 2023-11-30T21:49:27.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.634+0000 7fdff431e700 1 -- 172.21.15.49:0/679034575 wait complete. 2023-11-30T21:49:27.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.635+0000 7fdff431e700 1 Processor -- start 2023-11-30T21:49:27.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.635+0000 7fdff431e700 1 -- start start 2023-11-30T21:49:27.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.635+0000 7fdff431e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec0f6510 0x7fdfec106770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:27.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.635+0000 7fdff431e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec106c60 con 0x7fdfec0f6510 2023-11-30T21:49:27.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.635+0000 7fdff20ba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec0f6510 0x7fdfec106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:27.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.635+0000 7fdff20ba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec0f6510 0x7fdfec106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:48924/0 (socket says 172.21.15.49:48924) 2023-11-30T21:49:27.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.635+0000 7fdff20ba700 1 -- 172.21.15.49:0/1512091221 learned_addr learned my addr 172.21.15.49:0/1512091221 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:27.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.636+0000 7fdff20ba700 1 -- 172.21.15.49:0/1512091221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc009640 con 0x7fdfec0f6510 2023-11-30T21:49:27.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.636+0000 7fdff20ba700 1 --2- 172.21.15.49:0/1512091221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec0f6510 0x7fdfec106770 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fdfdc004a10 tx=0x7fdfdc004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:27.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.636+0000 7fdfeaffd700 1 -- 172.21.15.49:0/1512091221 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfdc018070 con 0x7fdfec0f6510 2023-11-30T21:49:27.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.636+0000 7fdff431e700 1 -- 172.21.15.49:0/1512091221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfec106e60 con 0x7fdfec0f6510 2023-11-30T21:49:27.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.636+0000 7fdfeaffd700 1 -- 172.21.15.49:0/1512091221 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfdc007e70 con 0x7fdfec0f6510 2023-11-30T21:49:27.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.636+0000 7fdfeaffd700 1 -- 172.21.15.49:0/1512091221 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfdc01d420 con 0x7fdfec0f6510 2023-11-30T21:49:27.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.636+0000 7fdff431e700 1 -- 172.21.15.49:0/1512091221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfec1072f0 con 0x7fdfec0f6510 2023-11-30T21:49:27.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.637+0000 7fdfeaffd700 1 -- 172.21.15.49:0/1512091221 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfdc013630 con 0x7fdfec0f6510 2023-11-30T21:49:27.640 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.637+0000 7fdff431e700 1 -- 172.21.15.49:0/1512091221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfec045ad0 con 0x7fdfec0f6510 2023-11-30T21:49:27.641 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.637+0000 7fdfeaffd700 1 --2- 172.21.15.49:0/1512091221 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdfd402ffb0 0x7fdfd4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:27.641 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.637+0000 7fdfeaffd700 1 -- 172.21.15.49:0/1512091221 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfdc0415f0 con 0x7fdfec0f6510 2023-11-30T21:49:27.641 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.637+0000 7fdff18b9700 1 --2- 172.21.15.49:0/1512091221 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdfd402ffb0 0x7fdfd4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:27.641 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.637+0000 7fdff18b9700 1 --2- 172.21.15.49:0/1512091221 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdfd402ffb0 0x7fdfd4032470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fdfe00096a0 tx=0x7fdfe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:27.642 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.641+0000 7fdfeaffd700 1 -- 172.21.15.49:0/1512091221 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfdc01a2d0 con 0x7fdfec0f6510 2023-11-30T21:49:27.860 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.859+0000 7fdff431e700 1 -- 172.21.15.49:0/1512091221 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfec0fbf60 con 0x7fdfec0f6510 2023-11-30T21:49:27.860 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.860+0000 7fdfeaffd700 1 -- 172.21.15.49:0/1512091221 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdfdc01a2d0 con 0x7fdfec0f6510 2023-11-30T21:49:27.861 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:27.861 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:27.862 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.862+0000 7fdff431e700 1 -- 172.21.15.49:0/1512091221 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdfd402ffb0 msgr2=0x7fdfd4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:27.862 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.862+0000 7fdff431e700 1 --2- 172.21.15.49:0/1512091221 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdfd402ffb0 0x7fdfd4032470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fdfe00096a0 tx=0x7fdfe0006b40).stop 2023-11-30T21:49:27.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.862+0000 7fdff431e700 1 -- 172.21.15.49:0/1512091221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec0f6510 msgr2=0x7fdfec106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:27.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.862+0000 7fdff431e700 1 --2- 172.21.15.49:0/1512091221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec0f6510 0x7fdfec106770 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fdfdc004a10 tx=0x7fdfdc004d10).stop 2023-11-30T21:49:27.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.862+0000 7fdff431e700 1 -- 172.21.15.49:0/1512091221 shutdown_connections 2023-11-30T21:49:27.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.862+0000 7fdff431e700 1 --2- 172.21.15.49:0/1512091221 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdfd402ffb0 0x7fdfd4032470 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:27.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.862+0000 7fdff431e700 1 --2- 172.21.15.49:0/1512091221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfec0f6510 0x7fdfec106770 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:27.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.862+0000 7fdff431e700 1 -- 172.21.15.49:0/1512091221 >> 172.21.15.49:0/1512091221 conn(0x7fdfec0f1c70 msgr2=0x7fdfec0f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:27.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.862+0000 7fdff431e700 1 -- 172.21.15.49:0/1512091221 shutdown_connections 2023-11-30T21:49:27.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:27.862+0000 7fdff431e700 1 -- 172.21.15.49:0/1512091221 wait complete. 2023-11-30T21:49:27.863 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:28.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:28 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:27.860489+0000 mon.smithi027 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.49:0/1512091221' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:29.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:29.274 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:29.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:27.509763+0000 mgr.smithi027.brrjcl (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:30.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:30 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:29.510012+0000 mgr.smithi027.brrjcl (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:30.567 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.566+0000 7fee6f8c8700 1 -- 172.21.15.49:0/4227174354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee680f5410 msgr2=0x7fee680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:30.567 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.566+0000 7fee6f8c8700 1 --2- 172.21.15.49:0/4227174354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee680f5410 0x7fee680f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fee58009fb0 tx=0x7fee58009b30).stop 2023-11-30T21:49:30.567 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.566+0000 7fee6f8c8700 1 -- 172.21.15.49:0/4227174354 shutdown_connections 2023-11-30T21:49:30.567 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.566+0000 7fee6f8c8700 1 --2- 172.21.15.49:0/4227174354 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee680f5410 0x7fee680f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:30.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.566+0000 7fee6f8c8700 1 -- 172.21.15.49:0/4227174354 >> 172.21.15.49:0/4227174354 conn(0x7fee680f0b70 msgr2=0x7fee680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:30.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.566+0000 7fee6f8c8700 1 -- 172.21.15.49:0/4227174354 shutdown_connections 2023-11-30T21:49:30.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.566+0000 7fee6f8c8700 1 -- 172.21.15.49:0/4227174354 wait complete. 2023-11-30T21:49:30.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.567+0000 7fee6f8c8700 1 Processor -- start 2023-11-30T21:49:30.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.567+0000 7fee6f8c8700 1 -- start start 2023-11-30T21:49:30.569 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.567+0000 7fee6f8c8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee680f5410 0x7fee680fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:30.569 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.567+0000 7fee6f8c8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee680fb820 con 0x7fee680f5410 2023-11-30T21:49:30.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.567+0000 7fee6d664700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee680f5410 0x7fee680fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:30.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.567+0000 7fee6d664700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee680f5410 0x7fee680fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:42698/0 (socket says 172.21.15.49:42698) 2023-11-30T21:49:30.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.567+0000 7fee6d664700 1 -- 172.21.15.49:0/1674417580 learned_addr learned my addr 172.21.15.49:0/1674417580 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:30.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.567+0000 7fee6d664700 1 -- 172.21.15.49:0/1674417580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee58005040 con 0x7fee680f5410 2023-11-30T21:49:30.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.567+0000 7fee6d664700 1 --2- 172.21.15.49:0/1674417580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee680f5410 0x7fee680fd3a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fee58013510 tx=0x7fee5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:30.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.568+0000 7fee5dffb700 1 -- 172.21.15.49:0/1674417580 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee58009320 con 0x7fee680f5410 2023-11-30T21:49:30.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.568+0000 7fee6f8c8700 1 -- 172.21.15.49:0/1674417580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee680fb9c0 con 0x7fee680f5410 2023-11-30T21:49:30.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.568+0000 7fee5dffb700 1 -- 172.21.15.49:0/1674417580 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee58013f00 con 0x7fee680f5410 2023-11-30T21:49:30.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.568+0000 7fee5dffb700 1 -- 172.21.15.49:0/1674417580 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee58014d90 con 0x7fee680f5410 2023-11-30T21:49:30.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.568+0000 7fee6f8c8700 1 -- 172.21.15.49:0/1674417580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee680fbe50 con 0x7fee680f5410 2023-11-30T21:49:30.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.569+0000 7fee5dffb700 1 -- 172.21.15.49:0/1674417580 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee58014420 con 0x7fee680f5410 2023-11-30T21:49:30.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.569+0000 7fee6f8c8700 1 -- 172.21.15.49:0/1674417580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee68059100 con 0x7fee680f5410 2023-11-30T21:49:30.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.569+0000 7fee5dffb700 1 --2- 172.21.15.49:0/1674417580 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fee5402ff60 0x7fee54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:30.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.569+0000 7fee5dffb700 1 -- 172.21.15.49:0/1674417580 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee58041110 con 0x7fee680f5410 2023-11-30T21:49:30.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.569+0000 7fee6ce63700 1 --2- 172.21.15.49:0/1674417580 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fee5402ff60 0x7fee54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:30.573 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.569+0000 7fee6ce63700 1 --2- 172.21.15.49:0/1674417580 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fee5402ff60 0x7fee54032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fee640096a0 tx=0x7fee64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:30.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.577+0000 7fee5dffb700 1 -- 172.21.15.49:0/1674417580 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee580095f0 con 0x7fee680f5410 2023-11-30T21:49:30.796 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.796+0000 7fee6f8c8700 1 -- 172.21.15.49:0/1674417580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee68045ad0 con 0x7fee680f5410 2023-11-30T21:49:30.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.796+0000 7fee5dffb700 1 -- 172.21.15.49:0/1674417580 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee5801db70 con 0x7fee680f5410 2023-11-30T21:49:30.797 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:30.797 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:30.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.798+0000 7fee6f8c8700 1 -- 172.21.15.49:0/1674417580 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fee5402ff60 msgr2=0x7fee54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:30.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.798+0000 7fee6f8c8700 1 --2- 172.21.15.49:0/1674417580 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fee5402ff60 0x7fee54032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fee640096a0 tx=0x7fee64006b40).stop 2023-11-30T21:49:30.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.798+0000 7fee6f8c8700 1 -- 172.21.15.49:0/1674417580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee680f5410 msgr2=0x7fee680fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:30.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.798+0000 7fee6f8c8700 1 --2- 172.21.15.49:0/1674417580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee680f5410 0x7fee680fd3a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fee58013510 tx=0x7fee5800b040).stop 2023-11-30T21:49:30.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.798+0000 7fee6f8c8700 1 -- 172.21.15.49:0/1674417580 shutdown_connections 2023-11-30T21:49:30.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.798+0000 7fee6f8c8700 1 --2- 172.21.15.49:0/1674417580 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fee5402ff60 0x7fee54032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:30.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.798+0000 7fee6f8c8700 1 --2- 172.21.15.49:0/1674417580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee680f5410 0x7fee680fd3a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:30.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.798+0000 7fee6f8c8700 1 -- 172.21.15.49:0/1674417580 >> 172.21.15.49:0/1674417580 conn(0x7fee680f0b70 msgr2=0x7fee6805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:30.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.798+0000 7fee6f8c8700 1 -- 172.21.15.49:0/1674417580 shutdown_connections 2023-11-30T21:49:30.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:30.798+0000 7fee6f8c8700 1 -- 172.21.15.49:0/1674417580 wait complete. 2023-11-30T21:49:30.799 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:31.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:30.796696+0000 mon.smithi027 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.49:0/1674417580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:32.253 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:32.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:32.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:32 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:31.510353+0000 mgr.smithi027.brrjcl (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:33.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.452+0000 7fc62ad95700 1 -- 172.21.15.49:0/736303369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc624060280 msgr2=0x7fc6240f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:33.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.452+0000 7fc62ad95700 1 --2- 172.21.15.49:0/736303369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc624060280 0x7fc6240f90f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fc618009fb0 tx=0x7fc618009b30).stop 2023-11-30T21:49:33.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.452+0000 7fc62ad95700 1 -- 172.21.15.49:0/736303369 shutdown_connections 2023-11-30T21:49:33.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.452+0000 7fc62ad95700 1 --2- 172.21.15.49:0/736303369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc624060280 0x7fc6240f90f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:33.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.452+0000 7fc62ad95700 1 -- 172.21.15.49:0/736303369 >> 172.21.15.49:0/736303369 conn(0x7fc6240f0b70 msgr2=0x7fc6240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:33.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.452+0000 7fc62ad95700 1 -- 172.21.15.49:0/736303369 shutdown_connections 2023-11-30T21:49:33.454 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.452+0000 7fc62ad95700 1 -- 172.21.15.49:0/736303369 wait complete. 2023-11-30T21:49:33.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.452+0000 7fc62ad95700 1 Processor -- start 2023-11-30T21:49:33.455 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.452+0000 7fc62ad95700 1 -- start start 2023-11-30T21:49:33.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.453+0000 7fc62ad95700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc624060280 0x7fc6241066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:33.456 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.453+0000 7fc62ad95700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc624106bd0 con 0x7fc624060280 2023-11-30T21:49:33.457 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.453+0000 7fc628b31700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc624060280 0x7fc6241066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:33.457 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.453+0000 7fc628b31700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc624060280 0x7fc6241066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:42730/0 (socket says 172.21.15.49:42730) 2023-11-30T21:49:33.457 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.453+0000 7fc628b31700 1 -- 172.21.15.49:0/3839827326 learned_addr learned my addr 172.21.15.49:0/3839827326 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:33.457 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.453+0000 7fc628b31700 1 -- 172.21.15.49:0/3839827326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc618005040 con 0x7fc624060280 2023-11-30T21:49:33.457 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.453+0000 7fc628b31700 1 --2- 172.21.15.49:0/3839827326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc624060280 0x7fc6241066e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fc618013510 tx=0x7fc61800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:33.457 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.454+0000 7fc6217fa700 1 -- 172.21.15.49:0/3839827326 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc618009320 con 0x7fc624060280 2023-11-30T21:49:33.457 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.454+0000 7fc6217fa700 1 -- 172.21.15.49:0/3839827326 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc618013f00 con 0x7fc624060280 2023-11-30T21:49:33.457 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.454+0000 7fc62ad95700 1 -- 172.21.15.49:0/3839827326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc624106dd0 con 0x7fc624060280 2023-11-30T21:49:33.457 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.454+0000 7fc6217fa700 1 -- 172.21.15.49:0/3839827326 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc618014dc0 con 0x7fc624060280 2023-11-30T21:49:33.458 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.454+0000 7fc62ad95700 1 -- 172.21.15.49:0/3839827326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc624107260 con 0x7fc624060280 2023-11-30T21:49:33.458 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.455+0000 7fc6217fa700 1 -- 172.21.15.49:0/3839827326 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc618014420 con 0x7fc624060280 2023-11-30T21:49:33.458 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.455+0000 7fc62ad95700 1 -- 172.21.15.49:0/3839827326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc624045ad0 con 0x7fc624060280 2023-11-30T21:49:33.458 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.455+0000 7fc6217fa700 1 --2- 172.21.15.49:0/3839827326 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc61402ff60 0x7fc614032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:33.458 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.455+0000 7fc6217fa700 1 -- 172.21.15.49:0/3839827326 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc618041050 con 0x7fc624060280 2023-11-30T21:49:33.458 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.455+0000 7fc623fff700 1 --2- 172.21.15.49:0/3839827326 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc61402ff60 0x7fc614032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:33.458 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.455+0000 7fc623fff700 1 --2- 172.21.15.49:0/3839827326 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc61402ff60 0x7fc614032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc6100096a0 tx=0x7fc610006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:33.461 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.459+0000 7fc6217fa700 1 -- 172.21.15.49:0/3839827326 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc61801cb10 con 0x7fc624060280 2023-11-30T21:49:33.681 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.679+0000 7fc62ad95700 1 -- 172.21.15.49:0/3839827326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc624108110 con 0x7fc624060280 2023-11-30T21:49:33.681 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.680+0000 7fc6217fa700 1 -- 172.21.15.49:0/3839827326 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc61801cb10 con 0x7fc624060280 2023-11-30T21:49:33.682 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:33.682 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:33.683 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.681+0000 7fc62ad95700 1 -- 172.21.15.49:0/3839827326 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc61402ff60 msgr2=0x7fc614032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:33.683 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.681+0000 7fc62ad95700 1 --2- 172.21.15.49:0/3839827326 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc61402ff60 0x7fc614032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc6100096a0 tx=0x7fc610006b40).stop 2023-11-30T21:49:33.683 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.681+0000 7fc62ad95700 1 -- 172.21.15.49:0/3839827326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc624060280 msgr2=0x7fc6241066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:33.683 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.681+0000 7fc62ad95700 1 --2- 172.21.15.49:0/3839827326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc624060280 0x7fc6241066e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fc618013510 tx=0x7fc61800b040).stop 2023-11-30T21:49:33.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.681+0000 7fc62ad95700 1 -- 172.21.15.49:0/3839827326 shutdown_connections 2023-11-30T21:49:33.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.681+0000 7fc62ad95700 1 --2- 172.21.15.49:0/3839827326 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc61402ff60 0x7fc614032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:33.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.681+0000 7fc62ad95700 1 --2- 172.21.15.49:0/3839827326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc624060280 0x7fc6241066e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:33.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.681+0000 7fc62ad95700 1 -- 172.21.15.49:0/3839827326 >> 172.21.15.49:0/3839827326 conn(0x7fc6240f0b70 msgr2=0x7fc6240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:33.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.682+0000 7fc62ad95700 1 -- 172.21.15.49:0/3839827326 shutdown_connections 2023-11-30T21:49:33.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:33.682+0000 7fc62ad95700 1 -- 172.21.15.49:0/3839827326 wait complete. 2023-11-30T21:49:33.684 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:34.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:34 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:33.510552+0000 mgr.smithi027.brrjcl (mgr.14162) 170 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:34.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:34 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:33.681255+0000 mon.smithi027 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.49:0/3839827326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:34.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:34 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:33.835045+0000 mon.smithi027 (mon.0) 236 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' 2023-11-30T21:49:34.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:34 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:49:33.836435+0000 mgr.smithi027.brrjcl (mgr.14162) 171 : cephadm [INF] Deploying daemon node-exporter.smithi027 on smithi027 2023-11-30T21:49:35.093 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:35.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:36.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.320+0000 7f8b51e49700 1 -- 172.21.15.49:0/4166584584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4c060220 msgr2=0x7f8b4c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:36.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.320+0000 7f8b51e49700 1 --2- 172.21.15.49:0/4166584584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4c060220 0x7f8b4c0f9150 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f8b3c009fb0 tx=0x7f8b3c009b30).stop 2023-11-30T21:49:36.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.320+0000 7f8b51e49700 1 -- 172.21.15.49:0/4166584584 shutdown_connections 2023-11-30T21:49:36.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.320+0000 7f8b51e49700 1 --2- 172.21.15.49:0/4166584584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4c060220 0x7f8b4c0f9150 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:36.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.320+0000 7f8b51e49700 1 -- 172.21.15.49:0/4166584584 >> 172.21.15.49:0/4166584584 conn(0x7f8b4c0f0b70 msgr2=0x7f8b4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:36.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.320+0000 7f8b51e49700 1 -- 172.21.15.49:0/4166584584 shutdown_connections 2023-11-30T21:49:36.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.320+0000 7f8b51e49700 1 -- 172.21.15.49:0/4166584584 wait complete. 2023-11-30T21:49:36.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.320+0000 7f8b51e49700 1 Processor -- start 2023-11-30T21:49:36.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.320+0000 7f8b51e49700 1 -- start start 2023-11-30T21:49:36.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.321+0000 7f8b51e49700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4c060220 0x7f8b4c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:36.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.321+0000 7f8b51e49700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c106bd0 con 0x7f8b4c060220 2023-11-30T21:49:36.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.321+0000 7f8b4b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4c060220 0x7f8b4c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:36.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.321+0000 7f8b4b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4c060220 0x7f8b4c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:42760/0 (socket says 172.21.15.49:42760) 2023-11-30T21:49:36.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.321+0000 7f8b4b7fe700 1 -- 172.21.15.49:0/2102203151 learned_addr learned my addr 172.21.15.49:0/2102203151 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:36.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.321+0000 7f8b4b7fe700 1 -- 172.21.15.49:0/2102203151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c005040 con 0x7f8b4c060220 2023-11-30T21:49:36.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.321+0000 7f8b4b7fe700 1 --2- 172.21.15.49:0/2102203151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4c060220 0x7f8b4c1066e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f8b3c013510 tx=0x7f8b3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:36.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.322+0000 7f8b33fff700 1 -- 172.21.15.49:0/2102203151 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b3c009320 con 0x7f8b4c060220 2023-11-30T21:49:36.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.322+0000 7f8b51e49700 1 -- 172.21.15.49:0/2102203151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b4c106dd0 con 0x7f8b4c060220 2023-11-30T21:49:36.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.322+0000 7f8b33fff700 1 -- 172.21.15.49:0/2102203151 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b3c013f00 con 0x7f8b4c060220 2023-11-30T21:49:36.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.322+0000 7f8b33fff700 1 -- 172.21.15.49:0/2102203151 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b3c014dc0 con 0x7f8b4c060220 2023-11-30T21:49:36.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.322+0000 7f8b51e49700 1 -- 172.21.15.49:0/2102203151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b4c107260 con 0x7f8b4c060220 2023-11-30T21:49:36.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.322+0000 7f8b33fff700 1 -- 172.21.15.49:0/2102203151 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b3c01d3d0 con 0x7f8b4c060220 2023-11-30T21:49:36.325 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.323+0000 7f8b33fff700 1 --2- 172.21.15.49:0/2102203151 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8b3402ff60 0x7f8b34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:36.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.323+0000 7f8b33fff700 1 -- 172.21.15.49:0/2102203151 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b3c0412f0 con 0x7f8b4c060220 2023-11-30T21:49:36.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.323+0000 7f8b51e49700 1 -- 172.21.15.49:0/2102203151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4c045ad0 con 0x7f8b4c060220 2023-11-30T21:49:36.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.323+0000 7f8b4affd700 1 --2- 172.21.15.49:0/2102203151 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8b3402ff60 0x7f8b34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:36.326 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.323+0000 7f8b4affd700 1 --2- 172.21.15.49:0/2102203151 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8b3402ff60 0x7f8b34032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f8b400096a0 tx=0x7f8b40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:36.329 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.328+0000 7f8b33fff700 1 -- 172.21.15.49:0/2102203151 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b3c012330 con 0x7f8b4c060220 2023-11-30T21:49:36.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.546+0000 7f8b51e49700 1 -- 172.21.15.49:0/2102203151 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b4c0fc040 con 0x7f8b4c060220 2023-11-30T21:49:36.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.546+0000 7f8b33fff700 1 -- 172.21.15.49:0/2102203151 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8b3c02b030 con 0x7f8b4c060220 2023-11-30T21:49:36.548 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:36.548 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:36.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.548+0000 7f8b51e49700 1 -- 172.21.15.49:0/2102203151 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8b3402ff60 msgr2=0x7f8b34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:36.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.548+0000 7f8b51e49700 1 --2- 172.21.15.49:0/2102203151 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8b3402ff60 0x7f8b34032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f8b400096a0 tx=0x7f8b40006b40).stop 2023-11-30T21:49:36.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.548+0000 7f8b51e49700 1 -- 172.21.15.49:0/2102203151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4c060220 msgr2=0x7f8b4c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:36.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.548+0000 7f8b51e49700 1 --2- 172.21.15.49:0/2102203151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4c060220 0x7f8b4c1066e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f8b3c013510 tx=0x7f8b3c00b040).stop 2023-11-30T21:49:36.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.548+0000 7f8b51e49700 1 -- 172.21.15.49:0/2102203151 shutdown_connections 2023-11-30T21:49:36.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.548+0000 7f8b51e49700 1 --2- 172.21.15.49:0/2102203151 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8b3402ff60 0x7f8b34032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:36.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.548+0000 7f8b51e49700 1 --2- 172.21.15.49:0/2102203151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b4c060220 0x7f8b4c1066e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:36.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.548+0000 7f8b51e49700 1 -- 172.21.15.49:0/2102203151 >> 172.21.15.49:0/2102203151 conn(0x7f8b4c0f0b70 msgr2=0x7f8b4c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:36.550 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.548+0000 7f8b51e49700 1 -- 172.21.15.49:0/2102203151 shutdown_connections 2023-11-30T21:49:36.551 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:36.548+0000 7f8b51e49700 1 -- 172.21.15.49:0/2102203151 wait complete. 2023-11-30T21:49:36.551 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:37.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:35.510851+0000 mgr.smithi027.brrjcl (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:37.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:36.547874+0000 mon.smithi027 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.49:0/2102203151' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:37.972 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:37.972 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:39.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:38 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:37.511197+0000 mgr.smithi027.brrjcl (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:39.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.205+0000 7fc2bc083700 1 -- 172.21.15.49:0/1632625677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2b40f5410 msgr2=0x7fc2b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:39.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.205+0000 7fc2bc083700 1 --2- 172.21.15.49:0/1632625677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2b40f5410 0x7fc2b40f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fc2b0009fb0 tx=0x7fc2b0009b30).stop 2023-11-30T21:49:39.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.205+0000 7fc2bc083700 1 -- 172.21.15.49:0/1632625677 shutdown_connections 2023-11-30T21:49:39.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.205+0000 7fc2bc083700 1 --2- 172.21.15.49:0/1632625677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2b40f5410 0x7fc2b40f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:39.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.205+0000 7fc2bc083700 1 -- 172.21.15.49:0/1632625677 >> 172.21.15.49:0/1632625677 conn(0x7fc2b40f0b70 msgr2=0x7fc2b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:39.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.206+0000 7fc2bc083700 1 -- 172.21.15.49:0/1632625677 shutdown_connections 2023-11-30T21:49:39.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.206+0000 7fc2bc083700 1 -- 172.21.15.49:0/1632625677 wait complete. 2023-11-30T21:49:39.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.206+0000 7fc2bc083700 1 Processor -- start 2023-11-30T21:49:39.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.206+0000 7fc2bc083700 1 -- start start 2023-11-30T21:49:39.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.206+0000 7fc2bc083700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2b40f5410 0x7fc2b4106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:39.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.206+0000 7fc2bc083700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b4106c10 con 0x7fc2b40f5410 2023-11-30T21:49:39.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.207+0000 7fc2b9e1f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2b40f5410 0x7fc2b4106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:39.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.207+0000 7fc2b9e1f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2b40f5410 0x7fc2b4106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:46204/0 (socket says 172.21.15.49:46204) 2023-11-30T21:49:39.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.207+0000 7fc2b9e1f700 1 -- 172.21.15.49:0/439635385 learned_addr learned my addr 172.21.15.49:0/439635385 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:39.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.207+0000 7fc2b9e1f700 1 -- 172.21.15.49:0/439635385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b0005040 con 0x7fc2b40f5410 2023-11-30T21:49:39.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.207+0000 7fc2b9e1f700 1 --2- 172.21.15.49:0/439635385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2b40f5410 0x7fc2b4106720 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fc2b0013510 tx=0x7fc2b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:39.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.207+0000 7fc2a67fc700 1 -- 172.21.15.49:0/439635385 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2b0009320 con 0x7fc2b40f5410 2023-11-30T21:49:39.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.207+0000 7fc2bc083700 1 -- 172.21.15.49:0/439635385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b4106e10 con 0x7fc2b40f5410 2023-11-30T21:49:39.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.207+0000 7fc2bc083700 1 -- 172.21.15.49:0/439635385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b41072a0 con 0x7fc2b40f5410 2023-11-30T21:49:39.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.207+0000 7fc2a67fc700 1 -- 172.21.15.49:0/439635385 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2b0013f00 con 0x7fc2b40f5410 2023-11-30T21:49:39.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.208+0000 7fc2a67fc700 1 -- 172.21.15.49:0/439635385 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2b0014ce0 con 0x7fc2b40f5410 2023-11-30T21:49:39.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.208+0000 7fc2a67fc700 1 -- 172.21.15.49:0/439635385 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2b0014420 con 0x7fc2b40f5410 2023-11-30T21:49:39.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.208+0000 7fc2bc083700 1 -- 172.21.15.49:0/439635385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29c004fa0 con 0x7fc2b40f5410 2023-11-30T21:49:39.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.208+0000 7fc2a67fc700 1 --2- 172.21.15.49:0/439635385 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc2a002ff60 0x7fc2a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:39.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.208+0000 7fc2a67fc700 1 -- 172.21.15.49:0/439635385 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2b0040ed0 con 0x7fc2b40f5410 2023-11-30T21:49:39.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.209+0000 7fc2b961e700 1 --2- 172.21.15.49:0/439635385 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc2a002ff60 0x7fc2a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:39.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.209+0000 7fc2b961e700 1 --2- 172.21.15.49:0/439635385 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc2a002ff60 0x7fc2a0032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fc2a80096a0 tx=0x7fc2a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:39.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.213+0000 7fc2a67fc700 1 -- 172.21.15.49:0/439635385 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2b0014e40 con 0x7fc2b40f5410 2023-11-30T21:49:39.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.432+0000 7fc2bc083700 1 -- 172.21.15.49:0/439635385 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc29c005670 con 0x7fc2b40f5410 2023-11-30T21:49:39.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.433+0000 7fc2a67fc700 1 -- 172.21.15.49:0/439635385 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc2b0012750 con 0x7fc2b40f5410 2023-11-30T21:49:39.435 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:39.435 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:39.436 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.434+0000 7fc2bc083700 1 -- 172.21.15.49:0/439635385 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc2a002ff60 msgr2=0x7fc2a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:39.436 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.434+0000 7fc2bc083700 1 --2- 172.21.15.49:0/439635385 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc2a002ff60 0x7fc2a0032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fc2a80096a0 tx=0x7fc2a8006b40).stop 2023-11-30T21:49:39.436 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.434+0000 7fc2bc083700 1 -- 172.21.15.49:0/439635385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2b40f5410 msgr2=0x7fc2b4106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:39.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.434+0000 7fc2bc083700 1 --2- 172.21.15.49:0/439635385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2b40f5410 0x7fc2b4106720 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fc2b0013510 tx=0x7fc2b000b040).stop 2023-11-30T21:49:39.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.434+0000 7fc2bc083700 1 -- 172.21.15.49:0/439635385 shutdown_connections 2023-11-30T21:49:39.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.434+0000 7fc2bc083700 1 --2- 172.21.15.49:0/439635385 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc2a002ff60 0x7fc2a0032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:39.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.434+0000 7fc2bc083700 1 --2- 172.21.15.49:0/439635385 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2b40f5410 0x7fc2b4106720 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:39.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.434+0000 7fc2bc083700 1 -- 172.21.15.49:0/439635385 >> 172.21.15.49:0/439635385 conn(0x7fc2b40f0b70 msgr2=0x7fc2b405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:39.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.435+0000 7fc2bc083700 1 -- 172.21.15.49:0/439635385 shutdown_connections 2023-11-30T21:49:39.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:39.435+0000 7fc2bc083700 1 -- 172.21.15.49:0/439635385 wait complete. 2023-11-30T21:49:39.437 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:40.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:40 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:39.434440+0000 mon.smithi027 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.49:0/439635385' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:40.857 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:40.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:41.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:41 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:39.511505+0000 mgr.smithi027.brrjcl (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:42.082 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.079+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/3199053200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd0060280 msgr2=0x7f6fd00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:42.082 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.079+0000 7f6fd4d2e700 1 --2- 172.21.15.49:0/3199053200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd0060280 0x7f6fd00f90f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f6fc4009fb0 tx=0x7f6fc4009b30).stop 2023-11-30T21:49:42.082 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.079+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/3199053200 shutdown_connections 2023-11-30T21:49:42.082 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.079+0000 7f6fd4d2e700 1 --2- 172.21.15.49:0/3199053200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd0060280 0x7f6fd00f90f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:42.082 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.079+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/3199053200 >> 172.21.15.49:0/3199053200 conn(0x7f6fd00f0b70 msgr2=0x7f6fd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:42.082 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.079+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/3199053200 shutdown_connections 2023-11-30T21:49:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.079+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/3199053200 wait complete. 2023-11-30T21:49:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.080+0000 7f6fd4d2e700 1 Processor -- start 2023-11-30T21:49:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.080+0000 7f6fd4d2e700 1 -- start start 2023-11-30T21:49:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.080+0000 7f6fd4d2e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd0060280 0x7f6fd01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.080+0000 7f6fd4d2e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd0106bd0 con 0x7f6fd0060280 2023-11-30T21:49:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.080+0000 7f6fce59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd0060280 0x7f6fd01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.080+0000 7f6fce59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd0060280 0x7f6fd01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:46214/0 (socket says 172.21.15.49:46214) 2023-11-30T21:49:42.084 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.080+0000 7f6fce59c700 1 -- 172.21.15.49:0/1322452245 learned_addr learned my addr 172.21.15.49:0/1322452245 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:42.084 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.081+0000 7f6fce59c700 1 -- 172.21.15.49:0/1322452245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc4005040 con 0x7f6fd0060280 2023-11-30T21:49:42.084 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.081+0000 7f6fce59c700 1 --2- 172.21.15.49:0/1322452245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd0060280 0x7f6fd01066e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f6fc4013510 tx=0x7f6fc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:42.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.081+0000 7f6fbeffd700 1 -- 172.21.15.49:0/1322452245 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fc4009320 con 0x7f6fd0060280 2023-11-30T21:49:42.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.081+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/1322452245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd0106dd0 con 0x7f6fd0060280 2023-11-30T21:49:42.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.081+0000 7f6fbeffd700 1 -- 172.21.15.49:0/1322452245 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6fc4013f00 con 0x7f6fd0060280 2023-11-30T21:49:42.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.081+0000 7f6fbeffd700 1 -- 172.21.15.49:0/1322452245 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fc4014dc0 con 0x7f6fd0060280 2023-11-30T21:49:42.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.081+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/1322452245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd0107260 con 0x7f6fd0060280 2023-11-30T21:49:42.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.082+0000 7f6fbeffd700 1 -- 172.21.15.49:0/1322452245 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6fc40095f0 con 0x7f6fd0060280 2023-11-30T21:49:42.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.082+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/1322452245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd0045ad0 con 0x7f6fd0060280 2023-11-30T21:49:42.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.082+0000 7f6fbeffd700 1 --2- 172.21.15.49:0/1322452245 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6fb802ff60 0x7f6fb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:42.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.082+0000 7f6fbeffd700 1 -- 172.21.15.49:0/1322452245 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6fc4041000 con 0x7f6fd0060280 2023-11-30T21:49:42.085 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.082+0000 7f6fcdd9b700 1 --2- 172.21.15.49:0/1322452245 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6fb802ff60 0x7f6fb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:42.086 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.083+0000 7f6fcdd9b700 1 --2- 172.21.15.49:0/1322452245 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6fb802ff60 0x7f6fb8032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f6fc00096a0 tx=0x7f6fc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:42.088 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.087+0000 7f6fbeffd700 1 -- 172.21.15.49:0/1322452245 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6fc4014420 con 0x7f6fd0060280 2023-11-30T21:49:42.305 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.304+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/1322452245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6fd0108110 con 0x7f6fd0060280 2023-11-30T21:49:42.306 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.304+0000 7f6fbeffd700 1 -- 172.21.15.49:0/1322452245 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6fc4010230 con 0x7f6fd0060280 2023-11-30T21:49:42.306 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:42.306 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:42.308 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.306+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/1322452245 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6fb802ff60 msgr2=0x7f6fb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:42.308 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.306+0000 7f6fd4d2e700 1 --2- 172.21.15.49:0/1322452245 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6fb802ff60 0x7f6fb8032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f6fc00096a0 tx=0x7f6fc0006b40).stop 2023-11-30T21:49:42.308 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.306+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/1322452245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd0060280 msgr2=0x7f6fd01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:42.308 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.306+0000 7f6fd4d2e700 1 --2- 172.21.15.49:0/1322452245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd0060280 0x7f6fd01066e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f6fc4013510 tx=0x7f6fc400b040).stop 2023-11-30T21:49:42.308 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.306+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/1322452245 shutdown_connections 2023-11-30T21:49:42.308 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.306+0000 7f6fd4d2e700 1 --2- 172.21.15.49:0/1322452245 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6fb802ff60 0x7f6fb8032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:42.308 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.306+0000 7f6fd4d2e700 1 --2- 172.21.15.49:0/1322452245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd0060280 0x7f6fd01066e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:42.308 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.306+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/1322452245 >> 172.21.15.49:0/1322452245 conn(0x7f6fd00f0b70 msgr2=0x7f6fd00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:42.308 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.306+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/1322452245 shutdown_connections 2023-11-30T21:49:42.308 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:42.306+0000 7f6fd4d2e700 1 -- 172.21.15.49:0/1322452245 wait complete. 2023-11-30T21:49:42.309 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:43.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:41.511837+0000 mgr.smithi027.brrjcl (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:43.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:42.306115+0000 mon.smithi027 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.49:0/1322452245' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:43.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:43.720 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:45.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.005+0000 7fd089e40700 1 -- 172.21.15.49:0/4008714948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd084060200 msgr2=0x7fd0840fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:45.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.005+0000 7fd089e40700 1 --2- 172.21.15.49:0/4008714948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd084060200 0x7fd0840fb570 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fd074009fb0 tx=0x7fd074009b30).stop 2023-11-30T21:49:45.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.005+0000 7fd089e40700 1 -- 172.21.15.49:0/4008714948 shutdown_connections 2023-11-30T21:49:45.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.005+0000 7fd089e40700 1 --2- 172.21.15.49:0/4008714948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd084060200 0x7fd0840fb570 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:45.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.005+0000 7fd089e40700 1 -- 172.21.15.49:0/4008714948 >> 172.21.15.49:0/4008714948 conn(0x7fd0840f0b70 msgr2=0x7fd0840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:45.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.005+0000 7fd089e40700 1 -- 172.21.15.49:0/4008714948 shutdown_connections 2023-11-30T21:49:45.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.005+0000 7fd089e40700 1 -- 172.21.15.49:0/4008714948 wait complete. 2023-11-30T21:49:45.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.006+0000 7fd089e40700 1 Processor -- start 2023-11-30T21:49:45.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.006+0000 7fd089e40700 1 -- start start 2023-11-30T21:49:45.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.006+0000 7fd089e40700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd084060200 0x7fd0841088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:45.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.006+0000 7fd089e40700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd084108dc0 con 0x7fd084060200 2023-11-30T21:49:45.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.006+0000 7fd0837fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd084060200 0x7fd0841088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:45.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.006+0000 7fd0837fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd084060200 0x7fd0841088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:46238/0 (socket says 172.21.15.49:46238) 2023-11-30T21:49:45.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.006+0000 7fd0837fe700 1 -- 172.21.15.49:0/3628081143 learned_addr learned my addr 172.21.15.49:0/3628081143 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:45.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.006+0000 7fd0837fe700 1 -- 172.21.15.49:0/3628081143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd074005040 con 0x7fd084060200 2023-11-30T21:49:45.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.007+0000 7fd0837fe700 1 --2- 172.21.15.49:0/3628081143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd084060200 0x7fd0841088d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fd074013510 tx=0x7fd07400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:45.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.007+0000 7fd07bfff700 1 -- 172.21.15.49:0/3628081143 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd074009320 con 0x7fd084060200 2023-11-30T21:49:45.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.007+0000 7fd07bfff700 1 -- 172.21.15.49:0/3628081143 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd074013f00 con 0x7fd084060200 2023-11-30T21:49:45.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.007+0000 7fd089e40700 1 -- 172.21.15.49:0/3628081143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd084108fc0 con 0x7fd084060200 2023-11-30T21:49:45.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.007+0000 7fd07bfff700 1 -- 172.21.15.49:0/3628081143 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd074014dc0 con 0x7fd084060200 2023-11-30T21:49:45.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.007+0000 7fd089e40700 1 -- 172.21.15.49:0/3628081143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd084109450 con 0x7fd084060200 2023-11-30T21:49:45.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.008+0000 7fd07bfff700 1 -- 172.21.15.49:0/3628081143 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd074014420 con 0x7fd084060200 2023-11-30T21:49:45.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.008+0000 7fd089e40700 1 -- 172.21.15.49:0/3628081143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd084045ad0 con 0x7fd084060200 2023-11-30T21:49:45.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.008+0000 7fd07bfff700 1 --2- 172.21.15.49:0/3628081143 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd07002ffb0 0x7fd070032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:45.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.008+0000 7fd07bfff700 1 -- 172.21.15.49:0/3628081143 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0740410b0 con 0x7fd084060200 2023-11-30T21:49:45.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.008+0000 7fd082ffd700 1 --2- 172.21.15.49:0/3628081143 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd07002ffb0 0x7fd070032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:45.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.009+0000 7fd082ffd700 1 --2- 172.21.15.49:0/3628081143 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd07002ffb0 0x7fd070032470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fd06c0096a0 tx=0x7fd06c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:45.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.013+0000 7fd07bfff700 1 -- 172.21.15.49:0/3628081143 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd07406d2b0 con 0x7fd084060200 2023-11-30T21:49:45.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.229+0000 7fd089e40700 1 -- 172.21.15.49:0/3628081143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd08410a300 con 0x7fd084060200 2023-11-30T21:49:45.231 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.229+0000 7fd07bfff700 1 -- 172.21.15.49:0/3628081143 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd0740095f0 con 0x7fd084060200 2023-11-30T21:49:45.231 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:45.231 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:45.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.231+0000 7fd089e40700 1 -- 172.21.15.49:0/3628081143 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd07002ffb0 msgr2=0x7fd070032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:45.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.231+0000 7fd089e40700 1 --2- 172.21.15.49:0/3628081143 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd07002ffb0 0x7fd070032470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fd06c0096a0 tx=0x7fd06c006b40).stop 2023-11-30T21:49:45.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.231+0000 7fd089e40700 1 -- 172.21.15.49:0/3628081143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd084060200 msgr2=0x7fd0841088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:45.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.231+0000 7fd089e40700 1 --2- 172.21.15.49:0/3628081143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd084060200 0x7fd0841088d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fd074013510 tx=0x7fd07400b040).stop 2023-11-30T21:49:45.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.231+0000 7fd089e40700 1 -- 172.21.15.49:0/3628081143 shutdown_connections 2023-11-30T21:49:45.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.231+0000 7fd089e40700 1 --2- 172.21.15.49:0/3628081143 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd07002ffb0 0x7fd070032470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:45.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.231+0000 7fd089e40700 1 --2- 172.21.15.49:0/3628081143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd084060200 0x7fd0841088d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:45.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.231+0000 7fd089e40700 1 -- 172.21.15.49:0/3628081143 >> 172.21.15.49:0/3628081143 conn(0x7fd0840f0b70 msgr2=0x7fd0840f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:45.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.231+0000 7fd089e40700 1 -- 172.21.15.49:0/3628081143 shutdown_connections 2023-11-30T21:49:45.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:45.231+0000 7fd089e40700 1 -- 172.21.15.49:0/3628081143 wait complete. 2023-11-30T21:49:45.234 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:45 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:43.512190+0000 mgr.smithi027.brrjcl (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:46.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:46 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:45.230737+0000 mon.smithi027 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.49:0/3628081143' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:46.634 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:46.635 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:47.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:47 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:45.512464+0000 mgr.smithi027.brrjcl (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:47.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.871+0000 7f0535ad1700 1 -- 172.21.15.49:0/259836165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05300f62a0 msgr2=0x7f05300f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:47.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.871+0000 7f0535ad1700 1 --2- 172.21.15.49:0/259836165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05300f62a0 0x7f05300f66c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f0524009fb0 tx=0x7f0524009b30).stop 2023-11-30T21:49:47.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.871+0000 7f0535ad1700 1 -- 172.21.15.49:0/259836165 shutdown_connections 2023-11-30T21:49:47.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.871+0000 7f0535ad1700 1 --2- 172.21.15.49:0/259836165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05300f62a0 0x7f05300f66c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:47.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.871+0000 7f0535ad1700 1 -- 172.21.15.49:0/259836165 >> 172.21.15.49:0/259836165 conn(0x7f05300f1aa0 msgr2=0x7f05300f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:47.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.871+0000 7f0535ad1700 1 -- 172.21.15.49:0/259836165 shutdown_connections 2023-11-30T21:49:47.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.871+0000 7f0535ad1700 1 -- 172.21.15.49:0/259836165 wait complete. 2023-11-30T21:49:47.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.872+0000 7f0535ad1700 1 Processor -- start 2023-11-30T21:49:47.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.872+0000 7f0535ad1700 1 -- start start 2023-11-30T21:49:47.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.872+0000 7f0535ad1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05300f62a0 0x7f05301064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:47.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.872+0000 7f0535ad1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05301069a0 con 0x7f05300f62a0 2023-11-30T21:49:47.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.872+0000 7f0534acf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05300f62a0 0x7f05301064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:47.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.873+0000 7f0534acf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05300f62a0 0x7f05301064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:46250/0 (socket says 172.21.15.49:46250) 2023-11-30T21:49:47.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.873+0000 7f0534acf700 1 -- 172.21.15.49:0/2946989488 learned_addr learned my addr 172.21.15.49:0/2946989488 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:47.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.873+0000 7f0534acf700 1 -- 172.21.15.49:0/2946989488 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0524005040 con 0x7f05300f62a0 2023-11-30T21:49:47.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.873+0000 7f0534acf700 1 --2- 172.21.15.49:0/2946989488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05300f62a0 0x7f05301064b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f0524009830 tx=0x7f052400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:47.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.873+0000 7f052d7fa700 1 -- 172.21.15.49:0/2946989488 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0524009320 con 0x7f05300f62a0 2023-11-30T21:49:47.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.873+0000 7f0535ad1700 1 -- 172.21.15.49:0/2946989488 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0530106ba0 con 0x7f05300f62a0 2023-11-30T21:49:47.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.873+0000 7f052d7fa700 1 -- 172.21.15.49:0/2946989488 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05240136c0 con 0x7f05300f62a0 2023-11-30T21:49:47.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.873+0000 7f052d7fa700 1 -- 172.21.15.49:0/2946989488 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f052401ee90 con 0x7f05300f62a0 2023-11-30T21:49:47.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.873+0000 7f0535ad1700 1 -- 172.21.15.49:0/2946989488 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0530107030 con 0x7f05300f62a0 2023-11-30T21:49:47.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.874+0000 7f052d7fa700 1 -- 172.21.15.49:0/2946989488 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0524014440 con 0x7f05300f62a0 2023-11-30T21:49:47.880 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.874+0000 7f052d7fa700 1 --2- 172.21.15.49:0/2946989488 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f052002ff10 0x7f05200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:47.881 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.874+0000 7f052d7fa700 1 -- 172.21.15.49:0/2946989488 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0524040fe0 con 0x7f05300f62a0 2023-11-30T21:49:47.881 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.874+0000 7f0535ad1700 1 -- 172.21.15.49:0/2946989488 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0530045ad0 con 0x7f05300f62a0 2023-11-30T21:49:47.881 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.874+0000 7f052ffff700 1 --2- 172.21.15.49:0/2946989488 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f052002ff10 0x7f05200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:47.881 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.875+0000 7f052ffff700 1 --2- 172.21.15.49:0/2946989488 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f052002ff10 0x7f05200323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f051c0096a0 tx=0x7f051c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:47.881 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:47.879+0000 7f052d7fa700 1 -- 172.21.15.49:0/2946989488 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05240273f0 con 0x7f05300f62a0 2023-11-30T21:49:48.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:48.095+0000 7f0535ad1700 1 -- 172.21.15.49:0/2946989488 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0530107d50 con 0x7f05300f62a0 2023-11-30T21:49:48.097 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:48.096+0000 7f052d7fa700 1 -- 172.21.15.49:0/2946989488 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f052401c320 con 0x7f05300f62a0 2023-11-30T21:49:48.098 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:48.098 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:48.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:48.098+0000 7f0535ad1700 1 -- 172.21.15.49:0/2946989488 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f052002ff10 msgr2=0x7f05200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:48.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:48.098+0000 7f0535ad1700 1 --2- 172.21.15.49:0/2946989488 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f052002ff10 0x7f05200323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f051c0096a0 tx=0x7f051c006b40).stop 2023-11-30T21:49:48.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:48.098+0000 7f0535ad1700 1 -- 172.21.15.49:0/2946989488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05300f62a0 msgr2=0x7f05301064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:48.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:48.098+0000 7f0535ad1700 1 --2- 172.21.15.49:0/2946989488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05300f62a0 0x7f05301064b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f0524009830 tx=0x7f052400b040).stop 2023-11-30T21:49:48.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:48.098+0000 7f0535ad1700 1 -- 172.21.15.49:0/2946989488 shutdown_connections 2023-11-30T21:49:48.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:48.098+0000 7f0535ad1700 1 --2- 172.21.15.49:0/2946989488 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f052002ff10 0x7f05200323d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:48.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:48.098+0000 7f0535ad1700 1 --2- 172.21.15.49:0/2946989488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05300f62a0 0x7f05301064b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:48.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:48.098+0000 7f0535ad1700 1 -- 172.21.15.49:0/2946989488 >> 172.21.15.49:0/2946989488 conn(0x7f05300f1aa0 msgr2=0x7f05300f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:48.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:48.098+0000 7f0535ad1700 1 -- 172.21.15.49:0/2946989488 shutdown_connections 2023-11-30T21:49:48.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:48.098+0000 7f0535ad1700 1 -- 172.21.15.49:0/2946989488 wait complete. 2023-11-30T21:49:48.100 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:48.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:48 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:48.097456+0000 mon.smithi027 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.49:0/2946989488' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:49.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:49.510 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:49.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:49 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:47.512803+0000 mgr.smithi027.brrjcl (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:50.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.810+0000 7f5065966700 1 -- 172.21.15.49:0/3136547117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600f59d0 msgr2=0x7f50600f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:50.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.810+0000 7f5065966700 1 --2- 172.21.15.49:0/3136547117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600f59d0 0x7f50600f5df0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f5050009fb0 tx=0x7f5050009b30).stop 2023-11-30T21:49:50.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.810+0000 7f5065966700 1 -- 172.21.15.49:0/3136547117 shutdown_connections 2023-11-30T21:49:50.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.810+0000 7f5065966700 1 --2- 172.21.15.49:0/3136547117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600f59d0 0x7f50600f5df0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:50.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.810+0000 7f5065966700 1 -- 172.21.15.49:0/3136547117 >> 172.21.15.49:0/3136547117 conn(0x7f50600f02e0 msgr2=0x7f50600f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:50.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.810+0000 7f5065966700 1 -- 172.21.15.49:0/3136547117 shutdown_connections 2023-11-30T21:49:50.813 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.810+0000 7f5065966700 1 -- 172.21.15.49:0/3136547117 wait complete. 2023-11-30T21:49:50.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.811+0000 7f5065966700 1 Processor -- start 2023-11-30T21:49:50.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.811+0000 7f5065966700 1 -- start start 2023-11-30T21:49:50.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.811+0000 7f5065966700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600f59d0 0x7f50600fd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:50.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.811+0000 7f5065966700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50600fb810 con 0x7f50600f59d0 2023-11-30T21:49:50.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.811+0000 7f505effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600f59d0 0x7f50600fd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:50.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.812+0000 7f505effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600f59d0 0x7f50600fd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:49080/0 (socket says 172.21.15.49:49080) 2023-11-30T21:49:50.814 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.812+0000 7f505effd700 1 -- 172.21.15.49:0/1929012183 learned_addr learned my addr 172.21.15.49:0/1929012183 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:50.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.812+0000 7f505effd700 1 -- 172.21.15.49:0/1929012183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5050005040 con 0x7f50600f59d0 2023-11-30T21:49:50.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.812+0000 7f505effd700 1 --2- 172.21.15.49:0/1929012183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600f59d0 0x7f50600fd390 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f5050013510 tx=0x7f505000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:50.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.812+0000 7f50577fe700 1 -- 172.21.15.49:0/1929012183 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5050009320 con 0x7f50600f59d0 2023-11-30T21:49:50.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.812+0000 7f50577fe700 1 -- 172.21.15.49:0/1929012183 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5050013f00 con 0x7f50600f59d0 2023-11-30T21:49:50.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.812+0000 7f5065966700 1 -- 172.21.15.49:0/1929012183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50600fb9b0 con 0x7f50600f59d0 2023-11-30T21:49:50.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.812+0000 7f50577fe700 1 -- 172.21.15.49:0/1929012183 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5050014d90 con 0x7f50600f59d0 2023-11-30T21:49:50.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.812+0000 7f5065966700 1 -- 172.21.15.49:0/1929012183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50600fbe40 con 0x7f50600f59d0 2023-11-30T21:49:50.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.813+0000 7f50577fe700 1 -- 172.21.15.49:0/1929012183 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50500095f0 con 0x7f50600f59d0 2023-11-30T21:49:50.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.813+0000 7f5065966700 1 -- 172.21.15.49:0/1929012183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5060059100 con 0x7f50600f59d0 2023-11-30T21:49:50.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.813+0000 7f50577fe700 1 --2- 172.21.15.49:0/1929012183 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f504c02ff60 0x7f504c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:50.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.813+0000 7f50577fe700 1 -- 172.21.15.49:0/1929012183 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5050040fe0 con 0x7f50600f59d0 2023-11-30T21:49:50.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.813+0000 7f505e7fc700 1 --2- 172.21.15.49:0/1929012183 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f504c02ff60 0x7f504c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:50.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.814+0000 7f505e7fc700 1 --2- 172.21.15.49:0/1929012183 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f504c02ff60 0x7f504c032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f50480096a0 tx=0x7f5048006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:50.819 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:50.818+0000 7f50577fe700 1 -- 172.21.15.49:0/1929012183 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f505001d7f0 con 0x7f50600f59d0 2023-11-30T21:49:51.036 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:51.035+0000 7f5065966700 1 -- 172.21.15.49:0/1929012183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50600fae50 con 0x7f50600f59d0 2023-11-30T21:49:51.037 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:51.035+0000 7f50577fe700 1 -- 172.21.15.49:0/1929012183 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5050010230 con 0x7f50600f59d0 2023-11-30T21:49:51.037 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:51.037 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:51.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:51.037+0000 7f5065966700 1 -- 172.21.15.49:0/1929012183 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f504c02ff60 msgr2=0x7f504c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:51.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:51.037+0000 7f5065966700 1 --2- 172.21.15.49:0/1929012183 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f504c02ff60 0x7f504c032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f50480096a0 tx=0x7f5048006b40).stop 2023-11-30T21:49:51.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:51.037+0000 7f5065966700 1 -- 172.21.15.49:0/1929012183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600f59d0 msgr2=0x7f50600fd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:51.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:51.037+0000 7f5065966700 1 --2- 172.21.15.49:0/1929012183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600f59d0 0x7f50600fd390 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f5050013510 tx=0x7f505000b040).stop 2023-11-30T21:49:51.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:51.037+0000 7f5065966700 1 -- 172.21.15.49:0/1929012183 shutdown_connections 2023-11-30T21:49:51.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:51.037+0000 7f5065966700 1 --2- 172.21.15.49:0/1929012183 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f504c02ff60 0x7f504c032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:51.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:51.037+0000 7f5065966700 1 --2- 172.21.15.49:0/1929012183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600f59d0 0x7f50600fd390 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:51.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:51.037+0000 7f5065966700 1 -- 172.21.15.49:0/1929012183 >> 172.21.15.49:0/1929012183 conn(0x7f50600f02e0 msgr2=0x7f50600f4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:51.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:51.037+0000 7f5065966700 1 -- 172.21.15.49:0/1929012183 shutdown_connections 2023-11-30T21:49:51.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:51.037+0000 7f5065966700 1 -- 172.21.15.49:0/1929012183 wait complete. 2023-11-30T21:49:51.040 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:51.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:49.513164+0000 mgr.smithi027.brrjcl (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:51.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:51.036911+0000 mon.smithi027 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.49:0/1929012183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:52.452 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:52.453 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:53.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:53 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:51.513522+0000 mgr.smithi027.brrjcl (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:53.667 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.665+0000 7f6c02cbf700 1 -- 172.21.15.49:0/3024081908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bfc0f5410 msgr2=0x7f6bfc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:53.667 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.665+0000 7f6c02cbf700 1 --2- 172.21.15.49:0/3024081908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bfc0f5410 0x7f6bfc0f5830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f6bf0009fb0 tx=0x7f6bf0009b30).stop 2023-11-30T21:49:53.667 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.665+0000 7f6c02cbf700 1 -- 172.21.15.49:0/3024081908 shutdown_connections 2023-11-30T21:49:53.667 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.665+0000 7f6c02cbf700 1 --2- 172.21.15.49:0/3024081908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bfc0f5410 0x7f6bfc0f5830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:53.667 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.665+0000 7f6c02cbf700 1 -- 172.21.15.49:0/3024081908 >> 172.21.15.49:0/3024081908 conn(0x7f6bfc0f0b70 msgr2=0x7f6bfc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:53.667 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.665+0000 7f6c02cbf700 1 -- 172.21.15.49:0/3024081908 shutdown_connections 2023-11-30T21:49:53.667 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.665+0000 7f6c02cbf700 1 -- 172.21.15.49:0/3024081908 wait complete. 2023-11-30T21:49:53.669 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.666+0000 7f6c02cbf700 1 Processor -- start 2023-11-30T21:49:53.669 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.666+0000 7f6c02cbf700 1 -- start start 2023-11-30T21:49:53.669 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.666+0000 7f6c02cbf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bfc0f5410 0x7f6bfc109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:53.669 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.666+0000 7f6c02cbf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bfc060220 con 0x7f6bfc0f5410 2023-11-30T21:49:53.669 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.667+0000 7f6c00a5b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bfc0f5410 0x7f6bfc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:53.669 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.667+0000 7f6c00a5b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bfc0f5410 0x7f6bfc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:49106/0 (socket says 172.21.15.49:49106) 2023-11-30T21:49:53.669 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.667+0000 7f6c00a5b700 1 -- 172.21.15.49:0/272413210 learned_addr learned my addr 172.21.15.49:0/272413210 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:53.669 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.667+0000 7f6c00a5b700 1 -- 172.21.15.49:0/272413210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf0005040 con 0x7f6bfc0f5410 2023-11-30T21:49:53.670 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.667+0000 7f6c00a5b700 1 --2- 172.21.15.49:0/272413210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bfc0f5410 0x7f6bfc109450 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f6bf0013510 tx=0x7f6bf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:53.670 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.667+0000 7f6bf97fa700 1 -- 172.21.15.49:0/272413210 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf0009320 con 0x7f6bfc0f5410 2023-11-30T21:49:53.670 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.667+0000 7f6c02cbf700 1 -- 172.21.15.49:0/272413210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bfc060420 con 0x7f6bfc0f5410 2023-11-30T21:49:53.670 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.667+0000 7f6bf97fa700 1 -- 172.21.15.49:0/272413210 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bf0013f00 con 0x7f6bfc0f5410 2023-11-30T21:49:53.670 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.667+0000 7f6bf97fa700 1 -- 172.21.15.49:0/272413210 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf0014d90 con 0x7f6bfc0f5410 2023-11-30T21:49:53.670 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.667+0000 7f6c02cbf700 1 -- 172.21.15.49:0/272413210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bfc109b00 con 0x7f6bfc0f5410 2023-11-30T21:49:53.671 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.668+0000 7f6bf97fa700 1 -- 172.21.15.49:0/272413210 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6bf0014420 con 0x7f6bfc0f5410 2023-11-30T21:49:53.671 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.669+0000 7f6bf97fa700 1 --2- 172.21.15.49:0/272413210 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6bec02ff60 0x7f6bec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:53.671 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.668+0000 7f6c02cbf700 1 -- 172.21.15.49:0/272413210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bfc059100 con 0x7f6bfc0f5410 2023-11-30T21:49:53.671 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.669+0000 7f6bf97fa700 1 -- 172.21.15.49:0/272413210 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6bf00410c0 con 0x7f6bfc0f5410 2023-11-30T21:49:53.671 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.669+0000 7f6bfbfff700 1 --2- 172.21.15.49:0/272413210 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6bec02ff60 0x7f6bec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:53.671 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.669+0000 7f6bfbfff700 1 --2- 172.21.15.49:0/272413210 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6bec02ff60 0x7f6bec032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6be80096a0 tx=0x7f6be8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:53.675 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.674+0000 7f6bf97fa700 1 -- 172.21.15.49:0/272413210 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bf006d2b0 con 0x7f6bfc0f5410 2023-11-30T21:49:53.891 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.890+0000 7f6c02cbf700 1 -- 172.21.15.49:0/272413210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6bfc045ad0 con 0x7f6bfc0f5410 2023-11-30T21:49:53.892 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.891+0000 7f6bf97fa700 1 -- 172.21.15.49:0/272413210 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6bf00095f0 con 0x7f6bfc0f5410 2023-11-30T21:49:53.892 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:53.893 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:53.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.892+0000 7f6c02cbf700 1 -- 172.21.15.49:0/272413210 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6bec02ff60 msgr2=0x7f6bec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:53.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.892+0000 7f6c02cbf700 1 --2- 172.21.15.49:0/272413210 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6bec02ff60 0x7f6bec032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f6be80096a0 tx=0x7f6be8006b40).stop 2023-11-30T21:49:53.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.892+0000 7f6c02cbf700 1 -- 172.21.15.49:0/272413210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bfc0f5410 msgr2=0x7f6bfc109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:53.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.892+0000 7f6c02cbf700 1 --2- 172.21.15.49:0/272413210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bfc0f5410 0x7f6bfc109450 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f6bf0013510 tx=0x7f6bf000b040).stop 2023-11-30T21:49:53.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.892+0000 7f6c02cbf700 1 -- 172.21.15.49:0/272413210 shutdown_connections 2023-11-30T21:49:53.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.892+0000 7f6c02cbf700 1 --2- 172.21.15.49:0/272413210 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6bec02ff60 0x7f6bec032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:53.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.892+0000 7f6c02cbf700 1 --2- 172.21.15.49:0/272413210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bfc0f5410 0x7f6bfc109450 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:53.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.892+0000 7f6c02cbf700 1 -- 172.21.15.49:0/272413210 >> 172.21.15.49:0/272413210 conn(0x7f6bfc0f0b70 msgr2=0x7f6bfc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:53.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.892+0000 7f6c02cbf700 1 -- 172.21.15.49:0/272413210 shutdown_connections 2023-11-30T21:49:53.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:53.892+0000 7f6c02cbf700 1 -- 172.21.15.49:0/272413210 wait complete. 2023-11-30T21:49:53.896 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:54.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:54 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:53.892103+0000 mon.smithi027 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.49:0/272413210' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:55.272 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:55.273 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:55.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:53.513872+0000 mgr.smithi027.brrjcl (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:56.552 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.549+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/638198172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f9c0f6510 msgr2=0x7f5f9c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:56.553 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.549+0000 7f5fa3d6f700 1 --2- 172.21.15.49:0/638198172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f9c0f6510 0x7f5f9c0f6930 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f5f98009fb0 tx=0x7f5f98009b30).stop 2023-11-30T21:49:56.553 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.549+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/638198172 shutdown_connections 2023-11-30T21:49:56.553 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.549+0000 7f5fa3d6f700 1 --2- 172.21.15.49:0/638198172 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f9c0f6510 0x7f5f9c0f6930 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:56.553 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.549+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/638198172 >> 172.21.15.49:0/638198172 conn(0x7f5f9c0f1c70 msgr2=0x7f5f9c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:56.553 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.549+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/638198172 shutdown_connections 2023-11-30T21:49:56.553 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.550+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/638198172 wait complete. 2023-11-30T21:49:56.553 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.550+0000 7f5fa3d6f700 1 Processor -- start 2023-11-30T21:49:56.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.550+0000 7f5fa3d6f700 1 -- start start 2023-11-30T21:49:56.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.550+0000 7f5fa3d6f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f9c0f6510 0x7f5f9c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:56.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.550+0000 7f5fa3d6f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c106c20 con 0x7f5f9c0f6510 2023-11-30T21:49:56.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.551+0000 7f5fa1b0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f9c0f6510 0x7f5f9c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:56.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.551+0000 7f5fa1b0b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f9c0f6510 0x7f5f9c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:49126/0 (socket says 172.21.15.49:49126) 2023-11-30T21:49:56.554 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.551+0000 7f5fa1b0b700 1 -- 172.21.15.49:0/1192867142 learned_addr learned my addr 172.21.15.49:0/1192867142 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:56.555 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.551+0000 7f5fa1b0b700 1 -- 172.21.15.49:0/1192867142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f98005040 con 0x7f5f9c0f6510 2023-11-30T21:49:56.555 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.551+0000 7f5fa1b0b700 1 --2- 172.21.15.49:0/1192867142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f9c0f6510 0x7f5f9c106730 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f5f98013510 tx=0x7f5f9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:56.555 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.551+0000 7f5f8e7fc700 1 -- 172.21.15.49:0/1192867142 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f98009320 con 0x7f5f9c0f6510 2023-11-30T21:49:56.555 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.551+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/1192867142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9c106e20 con 0x7f5f9c0f6510 2023-11-30T21:49:56.555 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.551+0000 7f5f8e7fc700 1 -- 172.21.15.49:0/1192867142 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f98013f00 con 0x7f5f9c0f6510 2023-11-30T21:49:56.555 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.551+0000 7f5f8e7fc700 1 -- 172.21.15.49:0/1192867142 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f98014d90 con 0x7f5f9c0f6510 2023-11-30T21:49:56.555 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.551+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/1192867142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9c1072b0 con 0x7f5f9c0f6510 2023-11-30T21:49:56.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.552+0000 7f5f8e7fc700 1 -- 172.21.15.49:0/1192867142 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f980263a0 con 0x7f5f9c0f6510 2023-11-30T21:49:56.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.552+0000 7f5f8e7fc700 1 --2- 172.21.15.49:0/1192867142 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5f8802ff60 0x7f5f88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:56.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.552+0000 7f5f8e7fc700 1 -- 172.21.15.49:0/1192867142 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f98041020 con 0x7f5f9c0f6510 2023-11-30T21:49:56.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.552+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/1192867142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9c045ad0 con 0x7f5f9c0f6510 2023-11-30T21:49:56.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.553+0000 7f5fa130a700 1 --2- 172.21.15.49:0/1192867142 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5f8802ff60 0x7f5f88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:56.557 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.553+0000 7f5fa130a700 1 --2- 172.21.15.49:0/1192867142 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5f8802ff60 0x7f5f88032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f5f900096a0 tx=0x7f5f90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:56.558 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.557+0000 7f5f8e7fc700 1 -- 172.21.15.49:0/1192867142 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f98012920 con 0x7f5f9c0f6510 2023-11-30T21:49:56.776 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.775+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/1192867142 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f9c0fc050 con 0x7f5f9c0f6510 2023-11-30T21:49:56.776 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.776+0000 7f5f8e7fc700 1 -- 172.21.15.49:0/1192867142 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5f9802b020 con 0x7f5f9c0f6510 2023-11-30T21:49:56.776 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:56.777 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:56.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.778+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/1192867142 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5f8802ff60 msgr2=0x7f5f88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:56.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.778+0000 7f5fa3d6f700 1 --2- 172.21.15.49:0/1192867142 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5f8802ff60 0x7f5f88032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f5f900096a0 tx=0x7f5f90006b40).stop 2023-11-30T21:49:56.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.778+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/1192867142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f9c0f6510 msgr2=0x7f5f9c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:56.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.778+0000 7f5fa3d6f700 1 --2- 172.21.15.49:0/1192867142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f9c0f6510 0x7f5f9c106730 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f5f98013510 tx=0x7f5f9800b040).stop 2023-11-30T21:49:56.779 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.778+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/1192867142 shutdown_connections 2023-11-30T21:49:56.779 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.778+0000 7f5fa3d6f700 1 --2- 172.21.15.49:0/1192867142 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5f8802ff60 0x7f5f88032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:56.779 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.778+0000 7f5fa3d6f700 1 --2- 172.21.15.49:0/1192867142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f9c0f6510 0x7f5f9c106730 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:56.779 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.778+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/1192867142 >> 172.21.15.49:0/1192867142 conn(0x7f5f9c0f1c70 msgr2=0x7f5f9c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:56.779 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.778+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/1192867142 shutdown_connections 2023-11-30T21:49:56.779 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:56.778+0000 7f5fa3d6f700 1 -- 172.21.15.49:0/1192867142 wait complete. 2023-11-30T21:49:56.779 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:49:57.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:57 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:55.514209+0000 mgr.smithi027.brrjcl (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:57.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:57 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:56.776387+0000 mon.smithi027 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.49:0/1192867142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:49:58.218 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:49:58.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:49:59.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.507+0000 7f774e27f700 1 -- 172.21.15.49:0/2515723128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77480f6510 msgr2=0x7f77480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:59.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.507+0000 7f774e27f700 1 --2- 172.21.15.49:0/2515723128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77480f6510 0x7f77480f6930 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f7738009fb0 tx=0x7f7738009b30).stop 2023-11-30T21:49:59.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.507+0000 7f774e27f700 1 -- 172.21.15.49:0/2515723128 shutdown_connections 2023-11-30T21:49:59.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.507+0000 7f774e27f700 1 --2- 172.21.15.49:0/2515723128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77480f6510 0x7f77480f6930 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:59.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.507+0000 7f774e27f700 1 -- 172.21.15.49:0/2515723128 >> 172.21.15.49:0/2515723128 conn(0x7f77480f1c70 msgr2=0x7f77480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:59.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.507+0000 7f774e27f700 1 -- 172.21.15.49:0/2515723128 shutdown_connections 2023-11-30T21:49:59.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.507+0000 7f774e27f700 1 -- 172.21.15.49:0/2515723128 wait complete. 2023-11-30T21:49:59.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.508+0000 7f774e27f700 1 Processor -- start 2023-11-30T21:49:59.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.508+0000 7f774e27f700 1 -- start start 2023-11-30T21:49:59.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.508+0000 7f774e27f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77480f6510 0x7f774806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:59.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.508+0000 7f774e27f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7748069e20 con 0x7f77480f6510 2023-11-30T21:49:59.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.508+0000 7f7747fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77480f6510 0x7f774806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:59.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.508+0000 7f7747fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77480f6510 0x7f774806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:33420/0 (socket says 172.21.15.49:33420) 2023-11-30T21:49:59.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.508+0000 7f7747fff700 1 -- 172.21.15.49:0/278922897 learned_addr learned my addr 172.21.15.49:0/278922897 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:49:59.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.508+0000 7f7747fff700 1 -- 172.21.15.49:0/278922897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7738005040 con 0x7f77480f6510 2023-11-30T21:49:59.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.509+0000 7f7747fff700 1 --2- 172.21.15.49:0/278922897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77480f6510 0x7f774806b9a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f7738013510 tx=0x7f773800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:59.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:49:59 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:57.514505+0000 mgr.smithi027.brrjcl (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:49:59.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.509+0000 7f7744ff9700 1 -- 172.21.15.49:0/278922897 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7738009320 con 0x7f77480f6510 2023-11-30T21:49:59.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.509+0000 7f774e27f700 1 -- 172.21.15.49:0/278922897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7748069fc0 con 0x7f77480f6510 2023-11-30T21:49:59.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.509+0000 7f7744ff9700 1 -- 172.21.15.49:0/278922897 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7738013f00 con 0x7f77480f6510 2023-11-30T21:49:59.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.509+0000 7f7744ff9700 1 -- 172.21.15.49:0/278922897 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7738014d90 con 0x7f77480f6510 2023-11-30T21:49:59.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.509+0000 7f774e27f700 1 -- 172.21.15.49:0/278922897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f774806a450 con 0x7f77480f6510 2023-11-30T21:49:59.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.510+0000 7f7744ff9700 1 -- 172.21.15.49:0/278922897 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77380095f0 con 0x7f77480f6510 2023-11-30T21:49:59.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.510+0000 7f774e27f700 1 -- 172.21.15.49:0/278922897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7748059100 con 0x7f77480f6510 2023-11-30T21:49:59.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.510+0000 7f7744ff9700 1 --2- 172.21.15.49:0/278922897 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f773402ff60 0x7f7734032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:49:59.512 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.510+0000 7f7744ff9700 1 -- 172.21.15.49:0/278922897 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7738041000 con 0x7f77480f6510 2023-11-30T21:49:59.513 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.510+0000 7f77477fe700 1 --2- 172.21.15.49:0/278922897 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f773402ff60 0x7f7734032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:49:59.513 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.510+0000 7f77477fe700 1 --2- 172.21.15.49:0/278922897 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f773402ff60 0x7f7734032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f77300096a0 tx=0x7f7730006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:49:59.515 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.515+0000 7f7744ff9700 1 -- 172.21.15.49:0/278922897 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7738014420 con 0x7f77480f6510 2023-11-30T21:49:59.734 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.733+0000 7f774e27f700 1 -- 172.21.15.49:0/278922897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77480fbf60 con 0x7f77480f6510 2023-11-30T21:49:59.734 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.734+0000 7f7744ff9700 1 -- 172.21.15.49:0/278922897 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f773801d7f0 con 0x7f77480f6510 2023-11-30T21:49:59.734 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:49:59.735 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:49:59.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.735+0000 7f774e27f700 1 -- 172.21.15.49:0/278922897 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f773402ff60 msgr2=0x7f7734032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:59.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.735+0000 7f774e27f700 1 --2- 172.21.15.49:0/278922897 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f773402ff60 0x7f7734032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f77300096a0 tx=0x7f7730006b40).stop 2023-11-30T21:49:59.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.735+0000 7f774e27f700 1 -- 172.21.15.49:0/278922897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77480f6510 msgr2=0x7f774806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:49:59.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.735+0000 7f774e27f700 1 --2- 172.21.15.49:0/278922897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77480f6510 0x7f774806b9a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f7738013510 tx=0x7f773800b040).stop 2023-11-30T21:49:59.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.735+0000 7f774e27f700 1 -- 172.21.15.49:0/278922897 shutdown_connections 2023-11-30T21:49:59.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.735+0000 7f774e27f700 1 --2- 172.21.15.49:0/278922897 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f773402ff60 0x7f7734032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:59.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.735+0000 7f774e27f700 1 --2- 172.21.15.49:0/278922897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77480f6510 0x7f774806b9a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:49:59.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.735+0000 7f774e27f700 1 -- 172.21.15.49:0/278922897 >> 172.21.15.49:0/278922897 conn(0x7f77480f1c70 msgr2=0x7f77480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:49:59.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.735+0000 7f774e27f700 1 -- 172.21.15.49:0/278922897 shutdown_connections 2023-11-30T21:49:59.737 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:49:59.735+0000 7f774e27f700 1 -- 172.21.15.49:0/278922897 wait complete. 2023-11-30T21:49:59.737 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:00.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:00 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:49:59.734205+0000 mon.smithi027 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.49:0/278922897' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:01.175 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:01.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:01.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:01 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:49:59.514734+0000 mgr.smithi027.brrjcl (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:02.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.403+0000 7fbf6ee79700 1 -- 172.21.15.49:0/2793649163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf680f6510 msgr2=0x7fbf680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:02.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.403+0000 7fbf6ee79700 1 --2- 172.21.15.49:0/2793649163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf680f6510 0x7fbf680f6930 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fbf5c009fb0 tx=0x7fbf5c009b30).stop 2023-11-30T21:50:02.410 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.403+0000 7fbf6ee79700 1 -- 172.21.15.49:0/2793649163 shutdown_connections 2023-11-30T21:50:02.410 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.403+0000 7fbf6ee79700 1 --2- 172.21.15.49:0/2793649163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf680f6510 0x7fbf680f6930 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:02.410 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.403+0000 7fbf6ee79700 1 -- 172.21.15.49:0/2793649163 >> 172.21.15.49:0/2793649163 conn(0x7fbf680f1c70 msgr2=0x7fbf680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:02.410 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.403+0000 7fbf6ee79700 1 -- 172.21.15.49:0/2793649163 shutdown_connections 2023-11-30T21:50:02.410 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.403+0000 7fbf6ee79700 1 -- 172.21.15.49:0/2793649163 wait complete. 2023-11-30T21:50:02.411 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.404+0000 7fbf6ee79700 1 Processor -- start 2023-11-30T21:50:02.411 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.404+0000 7fbf6ee79700 1 -- start start 2023-11-30T21:50:02.411 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.404+0000 7fbf6ee79700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf680f6510 0x7fbf68108930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:02.411 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.404+0000 7fbf6ee79700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf68108e20 con 0x7fbf680f6510 2023-11-30T21:50:02.411 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.405+0000 7fbf6cc15700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf680f6510 0x7fbf68108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:02.411 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.405+0000 7fbf6cc15700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf680f6510 0x7fbf68108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:33442/0 (socket says 172.21.15.49:33442) 2023-11-30T21:50:02.412 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.405+0000 7fbf6cc15700 1 -- 172.21.15.49:0/3477340049 learned_addr learned my addr 172.21.15.49:0/3477340049 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:02.412 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.405+0000 7fbf6cc15700 1 -- 172.21.15.49:0/3477340049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c005040 con 0x7fbf680f6510 2023-11-30T21:50:02.412 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.405+0000 7fbf6cc15700 1 --2- 172.21.15.49:0/3477340049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf680f6510 0x7fbf68108930 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fbf5c014000 tx=0x7fbf5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:02.412 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.405+0000 7fbf657fa700 1 -- 172.21.15.49:0/3477340049 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf5c0095f0 con 0x7fbf680f6510 2023-11-30T21:50:02.412 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.405+0000 7fbf657fa700 1 -- 172.21.15.49:0/3477340049 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf5c007e70 con 0x7fbf680f6510 2023-11-30T21:50:02.413 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.405+0000 7fbf6ee79700 1 -- 172.21.15.49:0/3477340049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf68109020 con 0x7fbf680f6510 2023-11-30T21:50:02.413 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.405+0000 7fbf657fa700 1 -- 172.21.15.49:0/3477340049 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf5c014ea0 con 0x7fbf680f6510 2023-11-30T21:50:02.413 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.405+0000 7fbf6ee79700 1 -- 172.21.15.49:0/3477340049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf681094b0 con 0x7fbf680f6510 2023-11-30T21:50:02.413 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.406+0000 7fbf657fa700 1 -- 172.21.15.49:0/3477340049 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbf5c025070 con 0x7fbf680f6510 2023-11-30T21:50:02.414 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.406+0000 7fbf6ee79700 1 -- 172.21.15.49:0/3477340049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf68045ad0 con 0x7fbf680f6510 2023-11-30T21:50:02.414 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.406+0000 7fbf657fa700 1 --2- 172.21.15.49:0/3477340049 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fbf5802ff60 0x7fbf58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:02.414 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.406+0000 7fbf657fa700 1 -- 172.21.15.49:0/3477340049 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbf5c040970 con 0x7fbf680f6510 2023-11-30T21:50:02.414 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.407+0000 7fbf67fff700 1 --2- 172.21.15.49:0/3477340049 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fbf5802ff60 0x7fbf58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:02.415 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.407+0000 7fbf67fff700 1 --2- 172.21.15.49:0/3477340049 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fbf5802ff60 0x7fbf58032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fbf540096a0 tx=0x7fbf54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:02.415 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.411+0000 7fbf657fa700 1 -- 172.21.15.49:0/3477340049 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf5c0079c0 con 0x7fbf680f6510 2023-11-30T21:50:02.634 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.633+0000 7fbf6ee79700 1 -- 172.21.15.49:0/3477340049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf680fbf60 con 0x7fbf680f6510 2023-11-30T21:50:02.635 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.634+0000 7fbf657fa700 1 -- 172.21.15.49:0/3477340049 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbf5c00f300 con 0x7fbf680f6510 2023-11-30T21:50:02.635 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:02.635 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:02.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.635+0000 7fbf6ee79700 1 -- 172.21.15.49:0/3477340049 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fbf5802ff60 msgr2=0x7fbf58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:02.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.636+0000 7fbf6ee79700 1 --2- 172.21.15.49:0/3477340049 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fbf5802ff60 0x7fbf58032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fbf540096a0 tx=0x7fbf54006b40).stop 2023-11-30T21:50:02.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.636+0000 7fbf6ee79700 1 -- 172.21.15.49:0/3477340049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf680f6510 msgr2=0x7fbf68108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:02.636 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.636+0000 7fbf6ee79700 1 --2- 172.21.15.49:0/3477340049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf680f6510 0x7fbf68108930 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fbf5c014000 tx=0x7fbf5c00b040).stop 2023-11-30T21:50:02.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.636+0000 7fbf6ee79700 1 -- 172.21.15.49:0/3477340049 shutdown_connections 2023-11-30T21:50:02.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.636+0000 7fbf6ee79700 1 --2- 172.21.15.49:0/3477340049 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fbf5802ff60 0x7fbf58032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:02.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.636+0000 7fbf6ee79700 1 --2- 172.21.15.49:0/3477340049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf680f6510 0x7fbf68108930 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:02.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.636+0000 7fbf6ee79700 1 -- 172.21.15.49:0/3477340049 >> 172.21.15.49:0/3477340049 conn(0x7fbf680f1c70 msgr2=0x7fbf680f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:02.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.636+0000 7fbf6ee79700 1 -- 172.21.15.49:0/3477340049 shutdown_connections 2023-11-30T21:50:02.637 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:02.636+0000 7fbf6ee79700 1 -- 172.21.15.49:0/3477340049 wait complete. 2023-11-30T21:50:02.637 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:03.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:03 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:01.515094+0000 mgr.smithi027.brrjcl (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:03.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:03 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:02.634686+0000 mon.smithi027 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.49:0/3477340049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:04.078 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:04.078 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:03.589351+0000 mon.smithi027 (mon.0) 247 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/mirror_snapshot_schedule"}]: dispatch 2023-11-30T21:50:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:03.595868+0000 mon.smithi027 (mon.0) 248 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/trash_purge_schedule"}]: dispatch 2023-11-30T21:50:05.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.280+0000 7fc904a17700 1 -- 172.21.15.49:0/701611250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9000601e0 msgr2=0x7fc9000f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:05.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.280+0000 7fc904a17700 1 --2- 172.21.15.49:0/701611250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9000601e0 0x7fc9000f8f20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fc8f0009fb0 tx=0x7fc8f0009b30).stop 2023-11-30T21:50:05.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.280+0000 7fc904a17700 1 -- 172.21.15.49:0/701611250 shutdown_connections 2023-11-30T21:50:05.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.280+0000 7fc904a17700 1 --2- 172.21.15.49:0/701611250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9000601e0 0x7fc9000f8f20 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:05.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.280+0000 7fc904a17700 1 -- 172.21.15.49:0/701611250 >> 172.21.15.49:0/701611250 conn(0x7fc9000f09a0 msgr2=0x7fc9000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:05.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.280+0000 7fc904a17700 1 -- 172.21.15.49:0/701611250 shutdown_connections 2023-11-30T21:50:05.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.280+0000 7fc904a17700 1 -- 172.21.15.49:0/701611250 wait complete. 2023-11-30T21:50:05.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.281+0000 7fc904a17700 1 Processor -- start 2023-11-30T21:50:05.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.281+0000 7fc904a17700 1 -- start start 2023-11-30T21:50:05.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.281+0000 7fc904a17700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9000601e0 0x7fc900106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:05.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.281+0000 7fc904a17700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9001069f0 con 0x7fc9000601e0 2023-11-30T21:50:05.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.281+0000 7fc8ff7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9000601e0 0x7fc900106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:05.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.281+0000 7fc8ff7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9000601e0 0x7fc900106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:33450/0 (socket says 172.21.15.49:33450) 2023-11-30T21:50:05.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.281+0000 7fc8ff7fe700 1 -- 172.21.15.49:0/2020905569 learned_addr learned my addr 172.21.15.49:0/2020905569 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:05.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.282+0000 7fc8ff7fe700 1 -- 172.21.15.49:0/2020905569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f0005040 con 0x7fc9000601e0 2023-11-30T21:50:05.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.282+0000 7fc8ff7fe700 1 --2- 172.21.15.49:0/2020905569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9000601e0 0x7fc900106500 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fc8f0013510 tx=0x7fc8f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:05.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.282+0000 7fc8e7fff700 1 -- 172.21.15.49:0/2020905569 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8f0009320 con 0x7fc9000601e0 2023-11-30T21:50:05.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.282+0000 7fc904a17700 1 -- 172.21.15.49:0/2020905569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc900106bf0 con 0x7fc9000601e0 2023-11-30T21:50:05.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.282+0000 7fc8e7fff700 1 -- 172.21.15.49:0/2020905569 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8f0013f00 con 0x7fc9000601e0 2023-11-30T21:50:05.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.282+0000 7fc8e7fff700 1 -- 172.21.15.49:0/2020905569 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8f0014d90 con 0x7fc9000601e0 2023-11-30T21:50:05.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.282+0000 7fc904a17700 1 -- 172.21.15.49:0/2020905569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc900107080 con 0x7fc9000601e0 2023-11-30T21:50:05.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.283+0000 7fc8e7fff700 1 -- 172.21.15.49:0/2020905569 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8f00095f0 con 0x7fc9000601e0 2023-11-30T21:50:05.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.283+0000 7fc904a17700 1 -- 172.21.15.49:0/2020905569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc900045ad0 con 0x7fc9000601e0 2023-11-30T21:50:05.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.283+0000 7fc8e7fff700 1 --2- 172.21.15.49:0/2020905569 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc8e802ff60 0x7fc8e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:05.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.283+0000 7fc8e7fff700 1 -- 172.21.15.49:0/2020905569 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc8f0040fc0 con 0x7fc9000601e0 2023-11-30T21:50:05.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.283+0000 7fc8feffd700 1 --2- 172.21.15.49:0/2020905569 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc8e802ff60 0x7fc8e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:05.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.284+0000 7fc8feffd700 1 --2- 172.21.15.49:0/2020905569 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc8e802ff60 0x7fc8e8032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fc8f40096a0 tx=0x7fc8f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:05.289 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.288+0000 7fc8e7fff700 1 -- 172.21.15.49:0/2020905569 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc8f0014420 con 0x7fc9000601e0 2023-11-30T21:50:05.508 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.507+0000 7fc904a17700 1 -- 172.21.15.49:0/2020905569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc900107ed0 con 0x7fc9000601e0 2023-11-30T21:50:05.509 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.508+0000 7fc8e7fff700 1 -- 172.21.15.49:0/2020905569 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc8f0010230 con 0x7fc9000601e0 2023-11-30T21:50:05.509 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:05.509 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:05.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.510+0000 7fc904a17700 1 -- 172.21.15.49:0/2020905569 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc8e802ff60 msgr2=0x7fc8e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:05.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.510+0000 7fc904a17700 1 --2- 172.21.15.49:0/2020905569 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc8e802ff60 0x7fc8e8032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fc8f40096a0 tx=0x7fc8f4006b40).stop 2023-11-30T21:50:05.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.510+0000 7fc904a17700 1 -- 172.21.15.49:0/2020905569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9000601e0 msgr2=0x7fc900106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:05.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.510+0000 7fc904a17700 1 --2- 172.21.15.49:0/2020905569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9000601e0 0x7fc900106500 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fc8f0013510 tx=0x7fc8f000b040).stop 2023-11-30T21:50:05.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.510+0000 7fc904a17700 1 -- 172.21.15.49:0/2020905569 shutdown_connections 2023-11-30T21:50:05.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.510+0000 7fc904a17700 1 --2- 172.21.15.49:0/2020905569 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc8e802ff60 0x7fc8e8032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:05.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.510+0000 7fc904a17700 1 --2- 172.21.15.49:0/2020905569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9000601e0 0x7fc900106500 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:05.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.510+0000 7fc904a17700 1 -- 172.21.15.49:0/2020905569 >> 172.21.15.49:0/2020905569 conn(0x7fc9000f09a0 msgr2=0x7fc9000f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:05.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.510+0000 7fc904a17700 1 -- 172.21.15.49:0/2020905569 shutdown_connections 2023-11-30T21:50:05.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:05.510+0000 7fc904a17700 1 -- 172.21.15.49:0/2020905569 wait complete. 2023-11-30T21:50:05.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:03.515478+0000 mgr.smithi027.brrjcl (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:05.512 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:06.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:06 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:05.508573+0000 mon.smithi027 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.49:0/2020905569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:06.941 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:06.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:07.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:07 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:05.515866+0000 mgr.smithi027.brrjcl (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:08.123 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.121+0000 7f616dc50700 1 -- 172.21.15.49:0/1387143555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6168060280 msgr2=0x7f61680f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:08.123 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.121+0000 7f616dc50700 1 --2- 172.21.15.49:0/1387143555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6168060280 0x7f61680f90f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f6158009fb0 tx=0x7f6158009b30).stop 2023-11-30T21:50:08.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.121+0000 7f616dc50700 1 -- 172.21.15.49:0/1387143555 shutdown_connections 2023-11-30T21:50:08.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.121+0000 7f616dc50700 1 --2- 172.21.15.49:0/1387143555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6168060280 0x7f61680f90f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:08.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.121+0000 7f616dc50700 1 -- 172.21.15.49:0/1387143555 >> 172.21.15.49:0/1387143555 conn(0x7f61680f0b70 msgr2=0x7f61680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:08.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.121+0000 7f616dc50700 1 -- 172.21.15.49:0/1387143555 shutdown_connections 2023-11-30T21:50:08.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.121+0000 7f616dc50700 1 -- 172.21.15.49:0/1387143555 wait complete. 2023-11-30T21:50:08.124 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.122+0000 7f616dc50700 1 Processor -- start 2023-11-30T21:50:08.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.122+0000 7f616dc50700 1 -- start start 2023-11-30T21:50:08.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.122+0000 7f616dc50700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6168060280 0x7f61680f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:08.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.122+0000 7f616dc50700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61680f4ff0 con 0x7f6168060280 2023-11-30T21:50:08.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.123+0000 7f61677fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6168060280 0x7f61680f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:08.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.123+0000 7f61677fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6168060280 0x7f61680f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:33472/0 (socket says 172.21.15.49:33472) 2023-11-30T21:50:08.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.123+0000 7f61677fe700 1 -- 172.21.15.49:0/1159045948 learned_addr learned my addr 172.21.15.49:0/1159045948 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:08.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.123+0000 7f61677fe700 1 -- 172.21.15.49:0/1159045948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6158005040 con 0x7f6168060280 2023-11-30T21:50:08.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.123+0000 7f61677fe700 1 --2- 172.21.15.49:0/1159045948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6168060280 0x7f61680f6b70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f6158013510 tx=0x7f615800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:08.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.123+0000 7f615ffff700 1 -- 172.21.15.49:0/1159045948 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6158009320 con 0x7f6168060280 2023-11-30T21:50:08.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.123+0000 7f616dc50700 1 -- 172.21.15.49:0/1159045948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61680f5190 con 0x7f6168060280 2023-11-30T21:50:08.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.123+0000 7f615ffff700 1 -- 172.21.15.49:0/1159045948 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6158013f00 con 0x7f6168060280 2023-11-30T21:50:08.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.123+0000 7f615ffff700 1 -- 172.21.15.49:0/1159045948 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6158014dc0 con 0x7f6168060280 2023-11-30T21:50:08.126 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.123+0000 7f616dc50700 1 -- 172.21.15.49:0/1159045948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61680f5620 con 0x7f6168060280 2023-11-30T21:50:08.127 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.124+0000 7f615ffff700 1 -- 172.21.15.49:0/1159045948 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61580095f0 con 0x7f6168060280 2023-11-30T21:50:08.127 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.124+0000 7f615ffff700 1 --2- 172.21.15.49:0/1159045948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f615402ffb0 0x7f6154032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:08.127 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.124+0000 7f615ffff700 1 -- 172.21.15.49:0/1159045948 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6158041000 con 0x7f6168060280 2023-11-30T21:50:08.127 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.124+0000 7f616dc50700 1 -- 172.21.15.49:0/1159045948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6168059100 con 0x7f6168060280 2023-11-30T21:50:08.127 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.124+0000 7f6166ffd700 1 --2- 172.21.15.49:0/1159045948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f615402ffb0 0x7f6154032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:08.128 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.125+0000 7f6166ffd700 1 --2- 172.21.15.49:0/1159045948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f615402ffb0 0x7f6154032470 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f61500096a0 tx=0x7f6150006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:08.131 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.129+0000 7f615ffff700 1 -- 172.21.15.49:0/1159045948 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6158014420 con 0x7f6168060280 2023-11-30T21:50:08.351 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.349+0000 7f616dc50700 1 -- 172.21.15.49:0/1159045948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61680f5c80 con 0x7f6168060280 2023-11-30T21:50:08.352 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.349+0000 7f615ffff700 1 -- 172.21.15.49:0/1159045948 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f615802b020 con 0x7f6168060280 2023-11-30T21:50:08.352 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:08.353 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:08.354 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.351+0000 7f616dc50700 1 -- 172.21.15.49:0/1159045948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f615402ffb0 msgr2=0x7f6154032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:08.354 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.351+0000 7f616dc50700 1 --2- 172.21.15.49:0/1159045948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f615402ffb0 0x7f6154032470 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f61500096a0 tx=0x7f6150006b40).stop 2023-11-30T21:50:08.354 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.351+0000 7f616dc50700 1 -- 172.21.15.49:0/1159045948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6168060280 msgr2=0x7f61680f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:08.354 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.351+0000 7f616dc50700 1 --2- 172.21.15.49:0/1159045948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6168060280 0x7f61680f6b70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f6158013510 tx=0x7f615800b040).stop 2023-11-30T21:50:08.354 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.351+0000 7f616dc50700 1 -- 172.21.15.49:0/1159045948 shutdown_connections 2023-11-30T21:50:08.355 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.351+0000 7f616dc50700 1 --2- 172.21.15.49:0/1159045948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f615402ffb0 0x7f6154032470 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:08.355 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.351+0000 7f616dc50700 1 --2- 172.21.15.49:0/1159045948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6168060280 0x7f61680f6b70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:08.355 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.351+0000 7f616dc50700 1 -- 172.21.15.49:0/1159045948 >> 172.21.15.49:0/1159045948 conn(0x7f61680f0b70 msgr2=0x7f61680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:08.355 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.351+0000 7f616dc50700 1 -- 172.21.15.49:0/1159045948 shutdown_connections 2023-11-30T21:50:08.355 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:08.351+0000 7f616dc50700 1 -- 172.21.15.49:0/1159045948 wait complete. 2023-11-30T21:50:08.355 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:09.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:09 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:07.516245+0000 mgr.smithi027.brrjcl (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:09.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:09 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:08.350746+0000 mon.smithi027 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.49:0/1159045948' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:09.734 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:09.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:11.045 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.042+0000 7f023155a700 1 -- 172.21.15.49:0/2360039437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0f51a0 msgr2=0x7f022c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:12.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.042+0000 7f023155a700 1 --2- 172.21.15.49:0/2360039437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0f51a0 0x7f022c0f55c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f021c009fb0 tx=0x7f021c009b30).stop 2023-11-30T21:50:12.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.042+0000 7f023155a700 1 -- 172.21.15.49:0/2360039437 shutdown_connections 2023-11-30T21:50:12.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.042+0000 7f023155a700 1 --2- 172.21.15.49:0/2360039437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0f51a0 0x7f022c0f55c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:12.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.042+0000 7f023155a700 1 -- 172.21.15.49:0/2360039437 >> 172.21.15.49:0/2360039437 conn(0x7f022c0f09a0 msgr2=0x7f022c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:12.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.042+0000 7f023155a700 1 -- 172.21.15.49:0/2360039437 shutdown_connections 2023-11-30T21:50:12.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.042+0000 7f023155a700 1 -- 172.21.15.49:0/2360039437 wait complete. 2023-11-30T21:50:12.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.042+0000 7f023155a700 1 Processor -- start 2023-11-30T21:50:12.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.043+0000 7f023155a700 1 -- start start 2023-11-30T21:50:12.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.043+0000 7f023155a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0f51a0 0x7f022c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:12.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.043+0000 7f023155a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c060240 con 0x7f022c0f51a0 2023-11-30T21:50:12.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.043+0000 7f022bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0f51a0 0x7f022c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:12.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.043+0000 7f022bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0f51a0 0x7f022c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:38622/0 (socket says 172.21.15.49:38622) 2023-11-30T21:50:12.812 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.043+0000 7f022bfff700 1 -- 172.21.15.49:0/3979909959 learned_addr learned my addr 172.21.15.49:0/3979909959 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:12.813 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:09.516583+0000 mgr.smithi027.brrjcl (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:12.813 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:12 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:11.267527+0000 mon.smithi027 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.49:0/3979909959' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:12.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.043+0000 7f022bfff700 1 -- 172.21.15.49:0/3979909959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c005040 con 0x7f022c0f51a0 2023-11-30T21:50:12.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.044+0000 7f022bfff700 1 --2- 172.21.15.49:0/3979909959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0f51a0 0x7f022c1091b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f021c013510 tx=0x7f021c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:12.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.044+0000 7f0228ff9700 1 -- 172.21.15.49:0/3979909959 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f021c009320 con 0x7f022c0f51a0 2023-11-30T21:50:12.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.044+0000 7f023155a700 1 -- 172.21.15.49:0/3979909959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022c060440 con 0x7f022c0f51a0 2023-11-30T21:50:12.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.044+0000 7f0228ff9700 1 -- 172.21.15.49:0/3979909959 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f021c013f00 con 0x7f022c0f51a0 2023-11-30T21:50:12.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.044+0000 7f0228ff9700 1 -- 172.21.15.49:0/3979909959 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f021c014dc0 con 0x7f022c0f51a0 2023-11-30T21:50:12.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.044+0000 7f023155a700 1 -- 172.21.15.49:0/3979909959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022c109860 con 0x7f022c0f51a0 2023-11-30T21:50:12.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.045+0000 7f0228ff9700 1 -- 172.21.15.49:0/3979909959 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f021c0095f0 con 0x7f022c0f51a0 2023-11-30T21:50:12.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.045+0000 7f023155a700 1 -- 172.21.15.49:0/3979909959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f022c045ad0 con 0x7f022c0f51a0 2023-11-30T21:50:12.815 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.045+0000 7f0228ff9700 1 --2- 172.21.15.49:0/3979909959 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f021402ff60 0x7f0214032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:12.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.045+0000 7f0228ff9700 1 -- 172.21.15.49:0/3979909959 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f021c041040 con 0x7f022c0f51a0 2023-11-30T21:50:12.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.045+0000 7f022b7fe700 1 --2- 172.21.15.49:0/3979909959 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f021402ff60 0x7f0214032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:12.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.045+0000 7f022b7fe700 1 --2- 172.21.15.49:0/3979909959 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f021402ff60 0x7f0214032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f02200096a0 tx=0x7f0220006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:12.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.049+0000 7f0228ff9700 1 -- 172.21.15.49:0/3979909959 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f021c014420 con 0x7f022c0f51a0 2023-11-30T21:50:12.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.265+0000 7f023155a700 1 -- 172.21.15.49:0/3979909959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f022c10a080 con 0x7f022c0f51a0 2023-11-30T21:50:12.816 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.266+0000 7f0228ff9700 1 -- 172.21.15.49:0/3979909959 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f021c010230 con 0x7f022c0f51a0 2023-11-30T21:50:12.844 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:12.852 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:12.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.268+0000 7f023155a700 1 -- 172.21.15.49:0/3979909959 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f021402ff60 msgr2=0x7f0214032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:12.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.268+0000 7f023155a700 1 --2- 172.21.15.49:0/3979909959 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f021402ff60 0x7f0214032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f02200096a0 tx=0x7f0220006b40).stop 2023-11-30T21:50:12.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.268+0000 7f023155a700 1 -- 172.21.15.49:0/3979909959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0f51a0 msgr2=0x7f022c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:12.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.268+0000 7f023155a700 1 --2- 172.21.15.49:0/3979909959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0f51a0 0x7f022c1091b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f021c013510 tx=0x7f021c00b040).stop 2023-11-30T21:50:12.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.268+0000 7f023155a700 1 -- 172.21.15.49:0/3979909959 shutdown_connections 2023-11-30T21:50:12.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.268+0000 7f023155a700 1 --2- 172.21.15.49:0/3979909959 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f021402ff60 0x7f0214032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:12.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.268+0000 7f023155a700 1 --2- 172.21.15.49:0/3979909959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0f51a0 0x7f022c1091b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:12.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.268+0000 7f023155a700 1 -- 172.21.15.49:0/3979909959 >> 172.21.15.49:0/3979909959 conn(0x7f022c0f09a0 msgr2=0x7f022c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:12.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.268+0000 7f023155a700 1 -- 172.21.15.49:0/3979909959 shutdown_connections 2023-11-30T21:50:12.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:11.268+0000 7f023155a700 1 -- 172.21.15.49:0/3979909959 wait complete. 2023-11-30T21:50:12.854 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:13.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:13 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:11.516836+0000 mgr.smithi027.brrjcl (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:13.855 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:13.855 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:15.117 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.116+0000 7f29a7e4c700 1 -- 172.21.15.49:0/2834188705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a00f5410 msgr2=0x7f29a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:15.118 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.116+0000 7f29a7e4c700 1 --2- 172.21.15.49:0/2834188705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a00f5410 0x7f29a00f5830 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f299c009fb0 tx=0x7f299c009b30).stop 2023-11-30T21:50:15.118 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.116+0000 7f29a7e4c700 1 -- 172.21.15.49:0/2834188705 shutdown_connections 2023-11-30T21:50:15.118 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.116+0000 7f29a7e4c700 1 --2- 172.21.15.49:0/2834188705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a00f5410 0x7f29a00f5830 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:15.118 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.116+0000 7f29a7e4c700 1 -- 172.21.15.49:0/2834188705 >> 172.21.15.49:0/2834188705 conn(0x7f29a00f0b70 msgr2=0x7f29a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:15.118 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.116+0000 7f29a7e4c700 1 -- 172.21.15.49:0/2834188705 shutdown_connections 2023-11-30T21:50:15.118 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.116+0000 7f29a7e4c700 1 -- 172.21.15.49:0/2834188705 wait complete. 2023-11-30T21:50:15.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.116+0000 7f29a7e4c700 1 Processor -- start 2023-11-30T21:50:15.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.117+0000 7f29a7e4c700 1 -- start start 2023-11-30T21:50:15.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.117+0000 7f29a7e4c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a00f5410 0x7f29a01093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:15.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.117+0000 7f29a7e4c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a0060280 con 0x7f29a00f5410 2023-11-30T21:50:15.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.117+0000 7f29a5be8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a00f5410 0x7f29a01093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:15.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.117+0000 7f29a5be8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a00f5410 0x7f29a01093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:38632/0 (socket says 172.21.15.49:38632) 2023-11-30T21:50:15.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.117+0000 7f29a5be8700 1 -- 172.21.15.49:0/673905561 learned_addr learned my addr 172.21.15.49:0/673905561 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:15.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.117+0000 7f29a5be8700 1 -- 172.21.15.49:0/673905561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c005040 con 0x7f29a00f5410 2023-11-30T21:50:15.120 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.118+0000 7f29a5be8700 1 --2- 172.21.15.49:0/673905561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a00f5410 0x7f29a01093f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f299c014000 tx=0x7f299c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:15.121 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.118+0000 7f29927fc700 1 -- 172.21.15.49:0/673905561 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f299c0095f0 con 0x7f29a00f5410 2023-11-30T21:50:15.121 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.118+0000 7f29a7e4c700 1 -- 172.21.15.49:0/673905561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a0060480 con 0x7f29a00f5410 2023-11-30T21:50:15.121 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.118+0000 7f29927fc700 1 -- 172.21.15.49:0/673905561 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f299c007e70 con 0x7f29a00f5410 2023-11-30T21:50:15.121 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.118+0000 7f29927fc700 1 -- 172.21.15.49:0/673905561 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f299c014ea0 con 0x7f29a00f5410 2023-11-30T21:50:15.121 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.118+0000 7f29a7e4c700 1 -- 172.21.15.49:0/673905561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a0109b20 con 0x7f29a00f5410 2023-11-30T21:50:15.122 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.119+0000 7f29927fc700 1 -- 172.21.15.49:0/673905561 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f299c014690 con 0x7f29a00f5410 2023-11-30T21:50:15.122 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.119+0000 7f29a7e4c700 1 -- 172.21.15.49:0/673905561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a0045ad0 con 0x7f29a00f5410 2023-11-30T21:50:15.122 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.119+0000 7f29927fc700 1 --2- 172.21.15.49:0/673905561 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f298c02ffb0 0x7f298c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:15.122 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.119+0000 7f29927fc700 1 -- 172.21.15.49:0/673905561 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f299c040830 con 0x7f29a00f5410 2023-11-30T21:50:15.122 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.119+0000 7f29a53e7700 1 --2- 172.21.15.49:0/673905561 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f298c02ffb0 0x7f298c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:15.122 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.120+0000 7f29a53e7700 1 --2- 172.21.15.49:0/673905561 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f298c02ffb0 0x7f298c032470 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f29940096a0 tx=0x7f2994006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:15.125 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.124+0000 7f29927fc700 1 -- 172.21.15.49:0/673905561 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f299c00f4e0 con 0x7f29a00f5410 2023-11-30T21:50:15.344 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.342+0000 7f29a7e4c700 1 -- 172.21.15.49:0/673905561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29a010a340 con 0x7f29a00f5410 2023-11-30T21:50:15.344 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.343+0000 7f29927fc700 1 -- 172.21.15.49:0/673905561 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f299c00f300 con 0x7f29a00f5410 2023-11-30T21:50:15.345 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:15.345 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:15.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.344+0000 7f29a7e4c700 1 -- 172.21.15.49:0/673905561 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f298c02ffb0 msgr2=0x7f298c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:15.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.344+0000 7f29a7e4c700 1 --2- 172.21.15.49:0/673905561 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f298c02ffb0 0x7f298c032470 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f29940096a0 tx=0x7f2994006b40).stop 2023-11-30T21:50:15.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.344+0000 7f29a7e4c700 1 -- 172.21.15.49:0/673905561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a00f5410 msgr2=0x7f29a01093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:15.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.344+0000 7f29a7e4c700 1 --2- 172.21.15.49:0/673905561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a00f5410 0x7f29a01093f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f299c014000 tx=0x7f299c00b040).stop 2023-11-30T21:50:15.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.344+0000 7f29a7e4c700 1 -- 172.21.15.49:0/673905561 shutdown_connections 2023-11-30T21:50:15.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.344+0000 7f29a7e4c700 1 --2- 172.21.15.49:0/673905561 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f298c02ffb0 0x7f298c032470 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:15.346 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.344+0000 7f29a7e4c700 1 --2- 172.21.15.49:0/673905561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29a00f5410 0x7f29a01093f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:15.347 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.344+0000 7f29a7e4c700 1 -- 172.21.15.49:0/673905561 >> 172.21.15.49:0/673905561 conn(0x7f29a00f0b70 msgr2=0x7f29a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:15.347 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.344+0000 7f29a7e4c700 1 -- 172.21.15.49:0/673905561 shutdown_connections 2023-11-30T21:50:15.347 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:15.344+0000 7f29a7e4c700 1 -- 172.21.15.49:0/673905561 wait complete. 2023-11-30T21:50:15.347 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:15.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:15 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:13.517131+0000 mgr.smithi027.brrjcl (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:16.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:16 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:15.344323+0000 mon.smithi027 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.49:0/673905561' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:16.804 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:16.805 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:17.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:15.517464+0000 mgr.smithi027.brrjcl (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:18.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.036+0000 7f36810aa700 1 -- 172.21.15.49:0/1384211001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f367c0f6510 msgr2=0x7f367c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:18.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.036+0000 7f36810aa700 1 --2- 172.21.15.49:0/1384211001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f367c0f6510 0x7f367c0f6930 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f366c009fb0 tx=0x7f366c009b30).stop 2023-11-30T21:50:18.038 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.036+0000 7f36810aa700 1 -- 172.21.15.49:0/1384211001 shutdown_connections 2023-11-30T21:50:18.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.036+0000 7f36810aa700 1 --2- 172.21.15.49:0/1384211001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f367c0f6510 0x7f367c0f6930 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:18.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.036+0000 7f36810aa700 1 -- 172.21.15.49:0/1384211001 >> 172.21.15.49:0/1384211001 conn(0x7f367c0f1c70 msgr2=0x7f367c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:18.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.036+0000 7f36810aa700 1 -- 172.21.15.49:0/1384211001 shutdown_connections 2023-11-30T21:50:18.039 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.036+0000 7f36810aa700 1 -- 172.21.15.49:0/1384211001 wait complete. 2023-11-30T21:50:18.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.037+0000 7f36810aa700 1 Processor -- start 2023-11-30T21:50:18.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.037+0000 7f36810aa700 1 -- start start 2023-11-30T21:50:18.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.037+0000 7f36810aa700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f367c0f6510 0x7f367c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:18.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.037+0000 7f36810aa700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c106c20 con 0x7f367c0f6510 2023-11-30T21:50:18.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.037+0000 7f367ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f367c0f6510 0x7f367c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:18.040 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.038+0000 7f367ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f367c0f6510 0x7f367c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:38650/0 (socket says 172.21.15.49:38650) 2023-11-30T21:50:18.041 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.038+0000 7f367ad9d700 1 -- 172.21.15.49:0/1231893854 learned_addr learned my addr 172.21.15.49:0/1231893854 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:18.041 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.038+0000 7f367ad9d700 1 -- 172.21.15.49:0/1231893854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c005040 con 0x7f367c0f6510 2023-11-30T21:50:18.041 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.038+0000 7f367ad9d700 1 --2- 172.21.15.49:0/1231893854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f367c0f6510 0x7f367c106730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f366c013510 tx=0x7f366c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:18.041 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.038+0000 7f36637fe700 1 -- 172.21.15.49:0/1231893854 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f366c009320 con 0x7f367c0f6510 2023-11-30T21:50:18.041 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.038+0000 7f36810aa700 1 -- 172.21.15.49:0/1231893854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367c106e20 con 0x7f367c0f6510 2023-11-30T21:50:18.041 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.038+0000 7f36637fe700 1 -- 172.21.15.49:0/1231893854 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f366c013f00 con 0x7f367c0f6510 2023-11-30T21:50:18.041 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.038+0000 7f36637fe700 1 -- 172.21.15.49:0/1231893854 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f366c014d90 con 0x7f367c0f6510 2023-11-30T21:50:18.041 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.038+0000 7f36810aa700 1 -- 172.21.15.49:0/1231893854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367c1072b0 con 0x7f367c0f6510 2023-11-30T21:50:18.042 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.039+0000 7f36637fe700 1 -- 172.21.15.49:0/1231893854 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f366c0263a0 con 0x7f367c0f6510 2023-11-30T21:50:18.042 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.039+0000 7f36810aa700 1 -- 172.21.15.49:0/1231893854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367c045ad0 con 0x7f367c0f6510 2023-11-30T21:50:18.042 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.039+0000 7f36637fe700 1 --2- 172.21.15.49:0/1231893854 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f366402ff60 0x7f3664032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:18.042 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.039+0000 7f36637fe700 1 -- 172.21.15.49:0/1231893854 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f366c0412d0 con 0x7f367c0f6510 2023-11-30T21:50:18.042 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.039+0000 7f367a59c700 1 --2- 172.21.15.49:0/1231893854 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f366402ff60 0x7f3664032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:18.042 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.040+0000 7f367a59c700 1 --2- 172.21.15.49:0/1231893854 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f366402ff60 0x7f3664032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f36700096a0 tx=0x7f3670006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:18.045 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.044+0000 7f36637fe700 1 -- 172.21.15.49:0/1231893854 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f366c01d420 con 0x7f367c0f6510 2023-11-30T21:50:18.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.263+0000 7f36810aa700 1 -- 172.21.15.49:0/1231893854 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f367c0fbf60 con 0x7f367c0f6510 2023-11-30T21:50:18.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.264+0000 7f36637fe700 1 -- 172.21.15.49:0/1231893854 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f366c01cb10 con 0x7f367c0f6510 2023-11-30T21:50:18.265 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:18.266 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:18.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.265+0000 7f36810aa700 1 -- 172.21.15.49:0/1231893854 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f366402ff60 msgr2=0x7f3664032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:18.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.265+0000 7f36810aa700 1 --2- 172.21.15.49:0/1231893854 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f366402ff60 0x7f3664032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f36700096a0 tx=0x7f3670006b40).stop 2023-11-30T21:50:18.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.265+0000 7f36810aa700 1 -- 172.21.15.49:0/1231893854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f367c0f6510 msgr2=0x7f367c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:18.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.265+0000 7f36810aa700 1 --2- 172.21.15.49:0/1231893854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f367c0f6510 0x7f367c106730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f366c013510 tx=0x7f366c00b040).stop 2023-11-30T21:50:18.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.265+0000 7f36810aa700 1 -- 172.21.15.49:0/1231893854 shutdown_connections 2023-11-30T21:50:18.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.265+0000 7f36810aa700 1 --2- 172.21.15.49:0/1231893854 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f366402ff60 0x7f3664032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:18.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.265+0000 7f36810aa700 1 --2- 172.21.15.49:0/1231893854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f367c0f6510 0x7f367c106730 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:18.268 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.265+0000 7f36810aa700 1 -- 172.21.15.49:0/1231893854 >> 172.21.15.49:0/1231893854 conn(0x7f367c0f1c70 msgr2=0x7f367c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:18.268 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.265+0000 7f36810aa700 1 -- 172.21.15.49:0/1231893854 shutdown_connections 2023-11-30T21:50:18.268 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:18.265+0000 7f36810aa700 1 -- 172.21.15.49:0/1231893854 wait complete. 2023-11-30T21:50:18.268 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:19.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:17.517784+0000 mgr.smithi027.brrjcl (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:19.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:18.265232+0000 mon.smithi027 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.49:0/1231893854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:19.733 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:19.734 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:20.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.995+0000 7f34168cd700 1 -- 172.21.15.49:0/2926107712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34100f7ae0 msgr2=0x7f34100f7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:20.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.995+0000 7f34168cd700 1 --2- 172.21.15.49:0/2926107712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34100f7ae0 0x7f34100f7f00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f3400009fb0 tx=0x7f3400009b30).stop 2023-11-30T21:50:20.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.995+0000 7f34168cd700 1 -- 172.21.15.49:0/2926107712 shutdown_connections 2023-11-30T21:50:20.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.995+0000 7f34168cd700 1 --2- 172.21.15.49:0/2926107712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34100f7ae0 0x7f34100f7f00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:20.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.995+0000 7f34168cd700 1 -- 172.21.15.49:0/2926107712 >> 172.21.15.49:0/2926107712 conn(0x7f34100f0260 msgr2=0x7f34100f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:20.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.996+0000 7f34168cd700 1 -- 172.21.15.49:0/2926107712 shutdown_connections 2023-11-30T21:50:20.998 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.996+0000 7f34168cd700 1 -- 172.21.15.49:0/2926107712 wait complete. 2023-11-30T21:50:20.999 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.996+0000 7f34168cd700 1 Processor -- start 2023-11-30T21:50:20.999 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.996+0000 7f34168cd700 1 -- start start 2023-11-30T21:50:20.999 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.996+0000 7f34168cd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34100f7ae0 0x7f34100f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:20.999 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.996+0000 7f34168cd700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34100f5830 con 0x7f34100f7ae0 2023-11-30T21:50:21.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.997+0000 7f340ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34100f7ae0 0x7f34100f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:21.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.997+0000 7f340ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34100f7ae0 0x7f34100f73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:51298/0 (socket says 172.21.15.49:51298) 2023-11-30T21:50:21.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.997+0000 7f340ffff700 1 -- 172.21.15.49:0/4212534701 learned_addr learned my addr 172.21.15.49:0/4212534701 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:21.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.997+0000 7f340ffff700 1 -- 172.21.15.49:0/4212534701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3400005040 con 0x7f34100f7ae0 2023-11-30T21:50:21.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.997+0000 7f340ffff700 1 --2- 172.21.15.49:0/4212534701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34100f7ae0 0x7f34100f73b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f3400013510 tx=0x7f340000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:21.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.997+0000 7f340cff9700 1 -- 172.21.15.49:0/4212534701 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3400009320 con 0x7f34100f7ae0 2023-11-30T21:50:21.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.997+0000 7f34168cd700 1 -- 172.21.15.49:0/4212534701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34100f59d0 con 0x7f34100f7ae0 2023-11-30T21:50:21.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.997+0000 7f340cff9700 1 -- 172.21.15.49:0/4212534701 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3400013f00 con 0x7f34100f7ae0 2023-11-30T21:50:21.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.997+0000 7f340cff9700 1 -- 172.21.15.49:0/4212534701 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3400014d90 con 0x7f34100f7ae0 2023-11-30T21:50:21.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.997+0000 7f34168cd700 1 -- 172.21.15.49:0/4212534701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34100f5e60 con 0x7f34100f7ae0 2023-11-30T21:50:21.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.998+0000 7f340cff9700 1 -- 172.21.15.49:0/4212534701 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34000095f0 con 0x7f34100f7ae0 2023-11-30T21:50:21.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.998+0000 7f34168cd700 1 -- 172.21.15.49:0/4212534701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3410059100 con 0x7f34100f7ae0 2023-11-30T21:50:21.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.998+0000 7f340cff9700 1 --2- 172.21.15.49:0/4212534701 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f33f802ff60 0x7f33f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:21.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.998+0000 7f340cff9700 1 -- 172.21.15.49:0/4212534701 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3400040fe0 con 0x7f34100f7ae0 2023-11-30T21:50:21.001 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.999+0000 7f340f7fe700 1 --2- 172.21.15.49:0/4212534701 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f33f802ff60 0x7f33f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:21.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:20.999+0000 7f340f7fe700 1 --2- 172.21.15.49:0/4212534701 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f33f802ff60 0x7f33f8032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f34040096a0 tx=0x7f3404006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:21.005 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:21.003+0000 7f340cff9700 1 -- 172.21.15.49:0/4212534701 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3400010230 con 0x7f34100f7ae0 2023-11-30T21:50:21.227 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:21.225+0000 7f34168cd700 1 -- 172.21.15.49:0/4212534701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3410045ad0 con 0x7f34100f7ae0 2023-11-30T21:50:21.228 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:21.226+0000 7f340cff9700 1 -- 172.21.15.49:0/4212534701 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3400010230 con 0x7f34100f7ae0 2023-11-30T21:50:21.228 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:21.228 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:21.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:21.228+0000 7f34168cd700 1 -- 172.21.15.49:0/4212534701 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f33f802ff60 msgr2=0x7f33f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:21.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:21.228+0000 7f34168cd700 1 --2- 172.21.15.49:0/4212534701 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f33f802ff60 0x7f33f8032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f34040096a0 tx=0x7f3404006b40).stop 2023-11-30T21:50:21.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:21.228+0000 7f34168cd700 1 -- 172.21.15.49:0/4212534701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34100f7ae0 msgr2=0x7f34100f73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:21.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:21.228+0000 7f34168cd700 1 --2- 172.21.15.49:0/4212534701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34100f7ae0 0x7f34100f73b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f3400013510 tx=0x7f340000b040).stop 2023-11-30T21:50:21.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:21.228+0000 7f34168cd700 1 -- 172.21.15.49:0/4212534701 shutdown_connections 2023-11-30T21:50:21.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:21.228+0000 7f34168cd700 1 --2- 172.21.15.49:0/4212534701 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f33f802ff60 0x7f33f8032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:21.231 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:21.228+0000 7f34168cd700 1 --2- 172.21.15.49:0/4212534701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34100f7ae0 0x7f34100f73b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:21.231 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:21.228+0000 7f34168cd700 1 -- 172.21.15.49:0/4212534701 >> 172.21.15.49:0/4212534701 conn(0x7f34100f0260 msgr2=0x7f34100f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:21.231 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:21.228+0000 7f34168cd700 1 -- 172.21.15.49:0/4212534701 shutdown_connections 2023-11-30T21:50:21.231 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:21.228+0000 7f34168cd700 1 -- 172.21.15.49:0/4212534701 wait complete. 2023-11-30T21:50:21.231 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:21.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:21 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:19.518113+0000 mgr.smithi027.brrjcl (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:22.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:22 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:21.227702+0000 mon.smithi027 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.49:0/4212534701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:22.634 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:22.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:23.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:23 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:21.518433+0000 mgr.smithi027.brrjcl (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:23.907 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.905+0000 7f4e196b4700 1 -- 172.21.15.49:0/517915746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e140f59d0 msgr2=0x7f4e140f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:23.907 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.905+0000 7f4e196b4700 1 --2- 172.21.15.49:0/517915746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e140f59d0 0x7f4e140f5df0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f4e04009fb0 tx=0x7f4e04009b30).stop 2023-11-30T21:50:23.907 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.905+0000 7f4e196b4700 1 -- 172.21.15.49:0/517915746 shutdown_connections 2023-11-30T21:50:23.907 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.905+0000 7f4e196b4700 1 --2- 172.21.15.49:0/517915746 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e140f59d0 0x7f4e140f5df0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:23.907 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.905+0000 7f4e196b4700 1 -- 172.21.15.49:0/517915746 >> 172.21.15.49:0/517915746 conn(0x7f4e140f02e0 msgr2=0x7f4e140f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:23.907 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.905+0000 7f4e196b4700 1 -- 172.21.15.49:0/517915746 shutdown_connections 2023-11-30T21:50:23.907 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.905+0000 7f4e196b4700 1 -- 172.21.15.49:0/517915746 wait complete. 2023-11-30T21:50:23.909 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.906+0000 7f4e196b4700 1 Processor -- start 2023-11-30T21:50:23.909 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.906+0000 7f4e196b4700 1 -- start start 2023-11-30T21:50:23.909 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.906+0000 7f4e196b4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e140f59d0 0x7f4e141093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:23.909 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.906+0000 7f4e196b4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e140f53b0 con 0x7f4e140f59d0 2023-11-30T21:50:23.909 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.906+0000 7f4e12ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e140f59d0 0x7f4e141093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:23.909 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.906+0000 7f4e12ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e140f59d0 0x7f4e141093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:51314/0 (socket says 172.21.15.49:51314) 2023-11-30T21:50:23.909 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.906+0000 7f4e12ffd700 1 -- 172.21.15.49:0/3468186838 learned_addr learned my addr 172.21.15.49:0/3468186838 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:23.909 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.907+0000 7f4e12ffd700 1 -- 172.21.15.49:0/3468186838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e04005040 con 0x7f4e140f59d0 2023-11-30T21:50:23.909 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.907+0000 7f4e12ffd700 1 --2- 172.21.15.49:0/3468186838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e140f59d0 0x7f4e141093c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f4e04013510 tx=0x7f4e0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:23.909 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.907+0000 7f4e0b7fe700 1 -- 172.21.15.49:0/3468186838 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e04009320 con 0x7f4e140f59d0 2023-11-30T21:50:23.910 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.907+0000 7f4e0b7fe700 1 -- 172.21.15.49:0/3468186838 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e04013f00 con 0x7f4e140f59d0 2023-11-30T21:50:23.910 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.907+0000 7f4e196b4700 1 -- 172.21.15.49:0/3468186838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e140f55b0 con 0x7f4e140f59d0 2023-11-30T21:50:23.910 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.907+0000 7f4e0b7fe700 1 -- 172.21.15.49:0/3468186838 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e04014d90 con 0x7f4e140f59d0 2023-11-30T21:50:23.910 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.907+0000 7f4e196b4700 1 -- 172.21.15.49:0/3468186838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e14109a70 con 0x7f4e140f59d0 2023-11-30T21:50:23.911 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.908+0000 7f4e0b7fe700 1 -- 172.21.15.49:0/3468186838 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e040095f0 con 0x7f4e140f59d0 2023-11-30T21:50:23.911 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.908+0000 7f4e196b4700 1 -- 172.21.15.49:0/3468186838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e14045ad0 con 0x7f4e140f59d0 2023-11-30T21:50:23.911 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.908+0000 7f4e0b7fe700 1 --2- 172.21.15.49:0/3468186838 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4e0002ff60 0x7f4e00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:23.911 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.908+0000 7f4e0b7fe700 1 -- 172.21.15.49:0/3468186838 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e04041000 con 0x7f4e140f59d0 2023-11-30T21:50:23.911 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.908+0000 7f4e127fc700 1 --2- 172.21.15.49:0/3468186838 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4e0002ff60 0x7f4e00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:23.911 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.909+0000 7f4e127fc700 1 --2- 172.21.15.49:0/3468186838 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4e0002ff60 0x7f4e00032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f4dfc0096a0 tx=0x7f4dfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:23.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:23.913+0000 7f4e0b7fe700 1 -- 172.21.15.49:0/3468186838 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e04014420 con 0x7f4e140f59d0 2023-11-30T21:50:24.133 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:24.131+0000 7f4e196b4700 1 -- 172.21.15.49:0/3468186838 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e1410a290 con 0x7f4e140f59d0 2023-11-30T21:50:24.134 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:24.132+0000 7f4e0b7fe700 1 -- 172.21.15.49:0/3468186838 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4e0401d7f0 con 0x7f4e140f59d0 2023-11-30T21:50:24.134 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:24.134 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:24.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:24.133+0000 7f4e196b4700 1 -- 172.21.15.49:0/3468186838 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4e0002ff60 msgr2=0x7f4e00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:24.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:24.133+0000 7f4e196b4700 1 --2- 172.21.15.49:0/3468186838 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4e0002ff60 0x7f4e00032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f4dfc0096a0 tx=0x7f4dfc006b40).stop 2023-11-30T21:50:24.135 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:24.133+0000 7f4e196b4700 1 -- 172.21.15.49:0/3468186838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e140f59d0 msgr2=0x7f4e141093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:24.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:24.133+0000 7f4e196b4700 1 --2- 172.21.15.49:0/3468186838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e140f59d0 0x7f4e141093c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f4e04013510 tx=0x7f4e0400b040).stop 2023-11-30T21:50:24.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:24.133+0000 7f4e196b4700 1 -- 172.21.15.49:0/3468186838 shutdown_connections 2023-11-30T21:50:24.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:24.133+0000 7f4e196b4700 1 --2- 172.21.15.49:0/3468186838 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4e0002ff60 0x7f4e00032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:24.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:24.133+0000 7f4e196b4700 1 --2- 172.21.15.49:0/3468186838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e140f59d0 0x7f4e141093c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:24.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:24.133+0000 7f4e196b4700 1 -- 172.21.15.49:0/3468186838 >> 172.21.15.49:0/3468186838 conn(0x7f4e140f02e0 msgr2=0x7f4e140f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:24.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:24.134+0000 7f4e196b4700 1 -- 172.21.15.49:0/3468186838 shutdown_connections 2023-11-30T21:50:24.136 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:24.134+0000 7f4e196b4700 1 -- 172.21.15.49:0/3468186838 wait complete. 2023-11-30T21:50:24.136 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:25.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:23.518751+0000 mgr.smithi027.brrjcl (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:25.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:24.133604+0000 mon.smithi027 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.49:0/3468186838' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:25.574 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:25.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:26.882 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.879+0000 7f5886b99700 1 -- 172.21.15.49:0/3363523890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58800f5410 msgr2=0x7f58800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:26.882 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.879+0000 7f5886b99700 1 --2- 172.21.15.49:0/3363523890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58800f5410 0x7f58800f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f5874009fb0 tx=0x7f5874009b30).stop 2023-11-30T21:50:26.882 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.879+0000 7f5886b99700 1 -- 172.21.15.49:0/3363523890 shutdown_connections 2023-11-30T21:50:26.882 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.879+0000 7f5886b99700 1 --2- 172.21.15.49:0/3363523890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58800f5410 0x7f58800f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:26.882 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.879+0000 7f5886b99700 1 -- 172.21.15.49:0/3363523890 >> 172.21.15.49:0/3363523890 conn(0x7f58800f0b70 msgr2=0x7f58800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:26.882 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.879+0000 7f5886b99700 1 -- 172.21.15.49:0/3363523890 shutdown_connections 2023-11-30T21:50:26.882 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.879+0000 7f5886b99700 1 -- 172.21.15.49:0/3363523890 wait complete. 2023-11-30T21:50:26.882 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.880+0000 7f5886b99700 1 Processor -- start 2023-11-30T21:50:26.883 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.880+0000 7f5886b99700 1 -- start start 2023-11-30T21:50:26.883 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.880+0000 7f5886b99700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58800f5410 0x7f5880105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:26.883 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.880+0000 7f5886b99700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5880105b20 con 0x7f58800f5410 2023-11-30T21:50:26.884 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.881+0000 7f5884935700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58800f5410 0x7f5880105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:26.884 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.881+0000 7f5884935700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58800f5410 0x7f5880105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:51330/0 (socket says 172.21.15.49:51330) 2023-11-30T21:50:26.885 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.881+0000 7f5884935700 1 -- 172.21.15.49:0/649577204 learned_addr learned my addr 172.21.15.49:0/649577204 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:26.885 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.881+0000 7f5884935700 1 -- 172.21.15.49:0/649577204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5874005040 con 0x7f58800f5410 2023-11-30T21:50:26.885 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.881+0000 7f5884935700 1 --2- 172.21.15.49:0/649577204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58800f5410 0x7f5880105630 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f5874013510 tx=0x7f587400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:26.885 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.881+0000 7f587d7fa700 1 -- 172.21.15.49:0/649577204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5874009320 con 0x7f58800f5410 2023-11-30T21:50:26.885 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.881+0000 7f587d7fa700 1 -- 172.21.15.49:0/649577204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5874013f00 con 0x7f58800f5410 2023-11-30T21:50:26.885 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.881+0000 7f5886b99700 1 -- 172.21.15.49:0/649577204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5880105d20 con 0x7f58800f5410 2023-11-30T21:50:26.885 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.881+0000 7f587d7fa700 1 -- 172.21.15.49:0/649577204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5874014d90 con 0x7f58800f5410 2023-11-30T21:50:26.885 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.881+0000 7f5886b99700 1 -- 172.21.15.49:0/649577204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58801061b0 con 0x7f58800f5410 2023-11-30T21:50:26.885 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.882+0000 7f587d7fa700 1 -- 172.21.15.49:0/649577204 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f58740095f0 con 0x7f58800f5410 2023-11-30T21:50:26.885 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.882+0000 7f5886b99700 1 -- 172.21.15.49:0/649577204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5880045ad0 con 0x7f58800f5410 2023-11-30T21:50:26.885 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.882+0000 7f587d7fa700 1 --2- 172.21.15.49:0/649577204 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f587002ff60 0x7f5870032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:26.886 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.882+0000 7f587d7fa700 1 -- 172.21.15.49:0/649577204 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5874041000 con 0x7f58800f5410 2023-11-30T21:50:26.886 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.883+0000 7f587ffff700 1 --2- 172.21.15.49:0/649577204 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f587002ff60 0x7f5870032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:26.886 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.883+0000 7f587ffff700 1 --2- 172.21.15.49:0/649577204 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f587002ff60 0x7f5870032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f586c0096a0 tx=0x7f586c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:26.889 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:26.887+0000 7f587d7fa700 1 -- 172.21.15.49:0/649577204 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f587401d7f0 con 0x7f58800f5410 2023-11-30T21:50:27.106 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:27.105+0000 7f5886b99700 1 -- 172.21.15.49:0/649577204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58800fae60 con 0x7f58800f5410 2023-11-30T21:50:27.107 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:27.105+0000 7f587d7fa700 1 -- 172.21.15.49:0/649577204 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5874010230 con 0x7f58800f5410 2023-11-30T21:50:27.107 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:27.107 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:27.108 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:27.107+0000 7f5886b99700 1 -- 172.21.15.49:0/649577204 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f587002ff60 msgr2=0x7f5870032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:27.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:27.107+0000 7f5886b99700 1 --2- 172.21.15.49:0/649577204 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f587002ff60 0x7f5870032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f586c0096a0 tx=0x7f586c006b40).stop 2023-11-30T21:50:27.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:27.107+0000 7f5886b99700 1 -- 172.21.15.49:0/649577204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58800f5410 msgr2=0x7f5880105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:27.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:27.107+0000 7f5886b99700 1 --2- 172.21.15.49:0/649577204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58800f5410 0x7f5880105630 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f5874013510 tx=0x7f587400b040).stop 2023-11-30T21:50:27.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:27.107+0000 7f5886b99700 1 -- 172.21.15.49:0/649577204 shutdown_connections 2023-11-30T21:50:27.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:27.107+0000 7f5886b99700 1 --2- 172.21.15.49:0/649577204 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f587002ff60 0x7f5870032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:27.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:27.107+0000 7f5886b99700 1 --2- 172.21.15.49:0/649577204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f58800f5410 0x7f5880105630 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:27.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:27.107+0000 7f5886b99700 1 -- 172.21.15.49:0/649577204 >> 172.21.15.49:0/649577204 conn(0x7f58800f0b70 msgr2=0x7f58800f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:27.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:27.107+0000 7f5886b99700 1 -- 172.21.15.49:0/649577204 shutdown_connections 2023-11-30T21:50:27.109 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:27.107+0000 7f5886b99700 1 -- 172.21.15.49:0/649577204 wait complete. 2023-11-30T21:50:27.110 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:27.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:27 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:25.519091+0000 mgr.smithi027.brrjcl (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:27.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:27 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:27.106826+0000 mon.smithi027 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.49:0/649577204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:28.503 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:28.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:29.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:27.519439+0000 mgr.smithi027.brrjcl (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:29.730 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.727+0000 7f7cdc165700 1 -- 172.21.15.49:0/3827235772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd40f5410 msgr2=0x7f7cd40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:29.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.727+0000 7f7cdc165700 1 --2- 172.21.15.49:0/3827235772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd40f5410 0x7f7cd40f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f7cd0009fb0 tx=0x7f7cd0009b30).stop 2023-11-30T21:50:29.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.727+0000 7f7cdc165700 1 -- 172.21.15.49:0/3827235772 shutdown_connections 2023-11-30T21:50:29.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.727+0000 7f7cdc165700 1 --2- 172.21.15.49:0/3827235772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd40f5410 0x7f7cd40f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:29.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.727+0000 7f7cdc165700 1 -- 172.21.15.49:0/3827235772 >> 172.21.15.49:0/3827235772 conn(0x7f7cd40f0b70 msgr2=0x7f7cd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:29.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.728+0000 7f7cdc165700 1 -- 172.21.15.49:0/3827235772 shutdown_connections 2023-11-30T21:50:29.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.728+0000 7f7cdc165700 1 -- 172.21.15.49:0/3827235772 wait complete. 2023-11-30T21:50:29.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.728+0000 7f7cdc165700 1 Processor -- start 2023-11-30T21:50:29.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.728+0000 7f7cdc165700 1 -- start start 2023-11-30T21:50:29.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.728+0000 7f7cdc165700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd40f5410 0x7f7cd4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:29.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.728+0000 7f7cdc165700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd4105b20 con 0x7f7cd40f5410 2023-11-30T21:50:29.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.729+0000 7f7cd9f01700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd40f5410 0x7f7cd4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:29.731 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.729+0000 7f7cd9f01700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd40f5410 0x7f7cd4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:38652/0 (socket says 172.21.15.49:38652) 2023-11-30T21:50:29.732 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.729+0000 7f7cd9f01700 1 -- 172.21.15.49:0/706907543 learned_addr learned my addr 172.21.15.49:0/706907543 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:29.732 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.729+0000 7f7cd9f01700 1 -- 172.21.15.49:0/706907543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd0005040 con 0x7f7cd40f5410 2023-11-30T21:50:29.732 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.729+0000 7f7cd9f01700 1 --2- 172.21.15.49:0/706907543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd40f5410 0x7f7cd4105630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f7cd0013510 tx=0x7f7cd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:29.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.729+0000 7f7cc67fc700 1 -- 172.21.15.49:0/706907543 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cd0009320 con 0x7f7cd40f5410 2023-11-30T21:50:29.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.729+0000 7f7cdc165700 1 -- 172.21.15.49:0/706907543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd4105d20 con 0x7f7cd40f5410 2023-11-30T21:50:29.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.729+0000 7f7cc67fc700 1 -- 172.21.15.49:0/706907543 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cd0013f00 con 0x7f7cd40f5410 2023-11-30T21:50:29.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.729+0000 7f7cc67fc700 1 -- 172.21.15.49:0/706907543 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cd0014d90 con 0x7f7cd40f5410 2023-11-30T21:50:29.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.729+0000 7f7cdc165700 1 -- 172.21.15.49:0/706907543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd41061b0 con 0x7f7cd40f5410 2023-11-30T21:50:29.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.730+0000 7f7cc67fc700 1 -- 172.21.15.49:0/706907543 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7cd00095f0 con 0x7f7cd40f5410 2023-11-30T21:50:29.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.730+0000 7f7cdc165700 1 -- 172.21.15.49:0/706907543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd4045ad0 con 0x7f7cd40f5410 2023-11-30T21:50:29.733 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.730+0000 7f7cc67fc700 1 --2- 172.21.15.49:0/706907543 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7cc002ff60 0x7f7cc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:29.734 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.730+0000 7f7cc67fc700 1 -- 172.21.15.49:0/706907543 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7cd0041000 con 0x7f7cd40f5410 2023-11-30T21:50:29.734 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.731+0000 7f7cd9700700 1 --2- 172.21.15.49:0/706907543 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7cc002ff60 0x7f7cc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:29.734 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.731+0000 7f7cd9700700 1 --2- 172.21.15.49:0/706907543 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7cc002ff60 0x7f7cc0032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f7cc80096a0 tx=0x7f7cc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:29.736 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.736+0000 7f7cc67fc700 1 -- 172.21.15.49:0/706907543 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7cd0014420 con 0x7f7cd40f5410 2023-11-30T21:50:29.953 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.952+0000 7f7cdc165700 1 -- 172.21.15.49:0/706907543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7cd40fae60 con 0x7f7cd40f5410 2023-11-30T21:50:29.954 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.953+0000 7f7cc67fc700 1 -- 172.21.15.49:0/706907543 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7cd0010230 con 0x7f7cd40f5410 2023-11-30T21:50:29.954 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:29.954 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.954+0000 7f7cdc165700 1 -- 172.21.15.49:0/706907543 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7cc002ff60 msgr2=0x7f7cc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.954+0000 7f7cdc165700 1 --2- 172.21.15.49:0/706907543 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7cc002ff60 0x7f7cc0032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f7cc80096a0 tx=0x7f7cc8006b40).stop 2023-11-30T21:50:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.954+0000 7f7cdc165700 1 -- 172.21.15.49:0/706907543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd40f5410 msgr2=0x7f7cd4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.954+0000 7f7cdc165700 1 --2- 172.21.15.49:0/706907543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd40f5410 0x7f7cd4105630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f7cd0013510 tx=0x7f7cd000b040).stop 2023-11-30T21:50:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.954+0000 7f7cdc165700 1 -- 172.21.15.49:0/706907543 shutdown_connections 2023-11-30T21:50:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.954+0000 7f7cdc165700 1 --2- 172.21.15.49:0/706907543 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f7cc002ff60 0x7f7cc0032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.954+0000 7f7cdc165700 1 --2- 172.21.15.49:0/706907543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd40f5410 0x7f7cd4105630 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.954+0000 7f7cdc165700 1 -- 172.21.15.49:0/706907543 >> 172.21.15.49:0/706907543 conn(0x7f7cd40f0b70 msgr2=0x7f7cd40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:29.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.954+0000 7f7cdc165700 1 -- 172.21.15.49:0/706907543 shutdown_connections 2023-11-30T21:50:29.957 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:29.954+0000 7f7cdc165700 1 -- 172.21.15.49:0/706907543 wait complete. 2023-11-30T21:50:29.957 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:30.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:30 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:29.954046+0000 mon.smithi027 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.49:0/706907543' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:31.372 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:31.372 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:31.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:29.519807+0000 mgr.smithi027.brrjcl (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:32.545 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.543+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/1363917801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf00f6510 msgr2=0x7f8bf00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:32.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.543+0000 7f8bf7ec7700 1 --2- 172.21.15.49:0/1363917801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf00f6510 0x7f8bf00f6930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f8be4009fb0 tx=0x7f8be4009b30).stop 2023-11-30T21:50:32.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.543+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/1363917801 shutdown_connections 2023-11-30T21:50:32.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.543+0000 7f8bf7ec7700 1 --2- 172.21.15.49:0/1363917801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf00f6510 0x7f8bf00f6930 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:32.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.543+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/1363917801 >> 172.21.15.49:0/1363917801 conn(0x7f8bf00f1c70 msgr2=0x7f8bf00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:32.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.543+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/1363917801 shutdown_connections 2023-11-30T21:50:32.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.543+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/1363917801 wait complete. 2023-11-30T21:50:32.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.544+0000 7f8bf7ec7700 1 Processor -- start 2023-11-30T21:50:32.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.544+0000 7f8bf7ec7700 1 -- start start 2023-11-30T21:50:32.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.544+0000 7f8bf7ec7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf00f6510 0x7f8bf0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:32.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.544+0000 7f8bf7ec7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf0106c20 con 0x7f8bf00f6510 2023-11-30T21:50:32.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.544+0000 7f8bf5c63700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf00f6510 0x7f8bf0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:32.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.544+0000 7f8bf5c63700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf00f6510 0x7f8bf0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:38680/0 (socket says 172.21.15.49:38680) 2023-11-30T21:50:32.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.544+0000 7f8bf5c63700 1 -- 172.21.15.49:0/2063593421 learned_addr learned my addr 172.21.15.49:0/2063593421 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:32.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.545+0000 7f8bf5c63700 1 -- 172.21.15.49:0/2063593421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be4005040 con 0x7f8bf00f6510 2023-11-30T21:50:32.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.545+0000 7f8bf5c63700 1 --2- 172.21.15.49:0/2063593421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf00f6510 0x7f8bf0106730 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f8be4013510 tx=0x7f8be400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:32.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.545+0000 7f8be27fc700 1 -- 172.21.15.49:0/2063593421 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8be4009320 con 0x7f8bf00f6510 2023-11-30T21:50:32.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.545+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/2063593421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf0106e20 con 0x7f8bf00f6510 2023-11-30T21:50:32.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.545+0000 7f8be27fc700 1 -- 172.21.15.49:0/2063593421 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8be4013f00 con 0x7f8bf00f6510 2023-11-30T21:50:32.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.545+0000 7f8be27fc700 1 -- 172.21.15.49:0/2063593421 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8be4014d90 con 0x7f8bf00f6510 2023-11-30T21:50:32.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.545+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/2063593421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf01072b0 con 0x7f8bf00f6510 2023-11-30T21:50:32.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.546+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/2063593421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf0045ad0 con 0x7f8bf00f6510 2023-11-30T21:50:32.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.546+0000 7f8be27fc700 1 -- 172.21.15.49:0/2063593421 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8be40095f0 con 0x7f8bf00f6510 2023-11-30T21:50:32.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.546+0000 7f8be27fc700 1 --2- 172.21.15.49:0/2063593421 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8bdc02ff60 0x7f8bdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:32.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.546+0000 7f8be27fc700 1 -- 172.21.15.49:0/2063593421 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8be4041000 con 0x7f8bf00f6510 2023-11-30T21:50:32.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.546+0000 7f8bf5462700 1 --2- 172.21.15.49:0/2063593421 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8bdc02ff60 0x7f8bdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:32.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.546+0000 7f8bf5462700 1 --2- 172.21.15.49:0/2063593421 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8bdc02ff60 0x7f8bdc032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8bec0096a0 tx=0x7f8bec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:32.551 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.550+0000 7f8be27fc700 1 -- 172.21.15.49:0/2063593421 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8be401d7f0 con 0x7f8bf00f6510 2023-11-30T21:50:32.769 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.768+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/2063593421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8bf00fbf60 con 0x7f8bf00f6510 2023-11-30T21:50:32.769 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.769+0000 7f8be27fc700 1 -- 172.21.15.49:0/2063593421 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8be4010230 con 0x7f8bf00f6510 2023-11-30T21:50:32.769 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:32.770 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:32.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.770+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/2063593421 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8bdc02ff60 msgr2=0x7f8bdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:32.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.770+0000 7f8bf7ec7700 1 --2- 172.21.15.49:0/2063593421 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8bdc02ff60 0x7f8bdc032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8bec0096a0 tx=0x7f8bec006b40).stop 2023-11-30T21:50:32.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.770+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/2063593421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf00f6510 msgr2=0x7f8bf0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:32.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.770+0000 7f8bf7ec7700 1 --2- 172.21.15.49:0/2063593421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf00f6510 0x7f8bf0106730 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f8be4013510 tx=0x7f8be400b040).stop 2023-11-30T21:50:32.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.770+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/2063593421 shutdown_connections 2023-11-30T21:50:32.771 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.770+0000 7f8bf7ec7700 1 --2- 172.21.15.49:0/2063593421 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8bdc02ff60 0x7f8bdc032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:32.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.770+0000 7f8bf7ec7700 1 --2- 172.21.15.49:0/2063593421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bf00f6510 0x7f8bf0106730 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:32.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.770+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/2063593421 >> 172.21.15.49:0/2063593421 conn(0x7f8bf00f1c70 msgr2=0x7f8bf00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:32.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.770+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/2063593421 shutdown_connections 2023-11-30T21:50:32.772 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:32.771+0000 7f8bf7ec7700 1 -- 172.21.15.49:0/2063593421 wait complete. 2023-11-30T21:50:32.772 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:33.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:33 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:31.520186+0000 mgr.smithi027.brrjcl (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:33.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:33 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:32.769240+0000 mon.smithi027 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.49:0/2063593421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:34.185 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:34.185 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:35.360 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.358+0000 7f863e837700 1 -- 172.21.15.49:0/790302259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86380f5410 msgr2=0x7f86380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:35.360 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.358+0000 7f863e837700 1 --2- 172.21.15.49:0/790302259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86380f5410 0x7f86380f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f8628009fb0 tx=0x7f8628009b30).stop 2023-11-30T21:50:35.360 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.358+0000 7f863e837700 1 -- 172.21.15.49:0/790302259 shutdown_connections 2023-11-30T21:50:35.360 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.358+0000 7f863e837700 1 --2- 172.21.15.49:0/790302259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86380f5410 0x7f86380f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:35.360 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.358+0000 7f863e837700 1 -- 172.21.15.49:0/790302259 >> 172.21.15.49:0/790302259 conn(0x7f86380f0b70 msgr2=0x7f86380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:35.360 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.358+0000 7f863e837700 1 -- 172.21.15.49:0/790302259 shutdown_connections 2023-11-30T21:50:35.360 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.358+0000 7f863e837700 1 -- 172.21.15.49:0/790302259 wait complete. 2023-11-30T21:50:35.360 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.359+0000 7f863e837700 1 Processor -- start 2023-11-30T21:50:35.360 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.359+0000 7f863e837700 1 -- start start 2023-11-30T21:50:35.361 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.359+0000 7f863e837700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86380f5410 0x7f8638108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:35.361 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.359+0000 7f863e837700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8638108e10 con 0x7f86380f5410 2023-11-30T21:50:35.362 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.360+0000 7f8637fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86380f5410 0x7f8638108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:35.362 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.360+0000 7f8637fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86380f5410 0x7f8638108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:38698/0 (socket says 172.21.15.49:38698) 2023-11-30T21:50:35.362 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.360+0000 7f8637fff700 1 -- 172.21.15.49:0/1142347390 learned_addr learned my addr 172.21.15.49:0/1142347390 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:35.362 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.360+0000 7f8637fff700 1 -- 172.21.15.49:0/1142347390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8628005040 con 0x7f86380f5410 2023-11-30T21:50:35.362 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.360+0000 7f8637fff700 1 --2- 172.21.15.49:0/1142347390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86380f5410 0x7f8638108920 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f8628013510 tx=0x7f862800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.360+0000 7f8634ff9700 1 -- 172.21.15.49:0/1142347390 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8628009320 con 0x7f86380f5410 2023-11-30T21:50:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.360+0000 7f8634ff9700 1 -- 172.21.15.49:0/1142347390 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8628013f00 con 0x7f86380f5410 2023-11-30T21:50:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.360+0000 7f863e837700 1 -- 172.21.15.49:0/1142347390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8638109010 con 0x7f86380f5410 2023-11-30T21:50:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.360+0000 7f8634ff9700 1 -- 172.21.15.49:0/1142347390 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8628014d90 con 0x7f86380f5410 2023-11-30T21:50:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.360+0000 7f863e837700 1 -- 172.21.15.49:0/1142347390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86381094a0 con 0x7f86380f5410 2023-11-30T21:50:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.361+0000 7f8634ff9700 1 -- 172.21.15.49:0/1142347390 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86280263a0 con 0x7f86380f5410 2023-11-30T21:50:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.361+0000 7f863e837700 1 -- 172.21.15.49:0/1142347390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8638045ad0 con 0x7f86380f5410 2023-11-30T21:50:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.361+0000 7f8634ff9700 1 --2- 172.21.15.49:0/1142347390 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f862002ff60 0x7f8620032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.362+0000 7f8634ff9700 1 -- 172.21.15.49:0/1142347390 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86280413b0 con 0x7f86380f5410 2023-11-30T21:50:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.362+0000 7f86377fe700 1 --2- 172.21.15.49:0/1142347390 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f862002ff60 0x7f8620032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:35.364 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.362+0000 7f86377fe700 1 --2- 172.21.15.49:0/1142347390 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f862002ff60 0x7f8620032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f862c0096a0 tx=0x7f862c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:35.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.367+0000 7f8634ff9700 1 -- 172.21.15.49:0/1142347390 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8628012510 con 0x7f86380f5410 2023-11-30T21:50:35.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:35 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:33.520452+0000 mgr.smithi027.brrjcl (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:35.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.585+0000 7f863e837700 1 -- 172.21.15.49:0/1142347390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f863810a350 con 0x7f86380f5410 2023-11-30T21:50:35.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.585+0000 7f8634ff9700 1 -- 172.21.15.49:0/1142347390 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f862800fbd0 con 0x7f86380f5410 2023-11-30T21:50:35.586 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:35.587 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:35.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.587+0000 7f863e837700 1 -- 172.21.15.49:0/1142347390 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f862002ff60 msgr2=0x7f8620032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:35.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.587+0000 7f863e837700 1 --2- 172.21.15.49:0/1142347390 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f862002ff60 0x7f8620032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f862c0096a0 tx=0x7f862c006b40).stop 2023-11-30T21:50:35.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.587+0000 7f863e837700 1 -- 172.21.15.49:0/1142347390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86380f5410 msgr2=0x7f8638108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:35.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.587+0000 7f863e837700 1 --2- 172.21.15.49:0/1142347390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86380f5410 0x7f8638108920 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f8628013510 tx=0x7f862800b040).stop 2023-11-30T21:50:35.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.587+0000 7f863e837700 1 -- 172.21.15.49:0/1142347390 shutdown_connections 2023-11-30T21:50:35.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.587+0000 7f863e837700 1 --2- 172.21.15.49:0/1142347390 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f862002ff60 0x7f8620032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:35.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.587+0000 7f863e837700 1 --2- 172.21.15.49:0/1142347390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86380f5410 0x7f8638108920 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:35.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.587+0000 7f863e837700 1 -- 172.21.15.49:0/1142347390 >> 172.21.15.49:0/1142347390 conn(0x7f86380f0b70 msgr2=0x7f863805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:35.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.587+0000 7f863e837700 1 -- 172.21.15.49:0/1142347390 shutdown_connections 2023-11-30T21:50:35.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:35.587+0000 7f863e837700 1 -- 172.21.15.49:0/1142347390 wait complete. 2023-11-30T21:50:35.588 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:36.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:35.585968+0000 mon.smithi027 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.49:0/1142347390' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:37.017 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:37.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:37.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:37 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:35.520789+0000 mgr.smithi027.brrjcl (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:38.263 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.263+0000 7ff5dacd0700 1 -- 172.21.15.49:0/3531044023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5d40f51a0 msgr2=0x7ff5d40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:38.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.263+0000 7ff5dacd0700 1 --2- 172.21.15.49:0/3531044023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5d40f51a0 0x7ff5d40f55c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7ff5d0009fb0 tx=0x7ff5d0009b30).stop 2023-11-30T21:50:38.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.263+0000 7ff5dacd0700 1 -- 172.21.15.49:0/3531044023 shutdown_connections 2023-11-30T21:50:38.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.263+0000 7ff5dacd0700 1 --2- 172.21.15.49:0/3531044023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5d40f51a0 0x7ff5d40f55c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:38.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.263+0000 7ff5dacd0700 1 -- 172.21.15.49:0/3531044023 >> 172.21.15.49:0/3531044023 conn(0x7ff5d40f09a0 msgr2=0x7ff5d40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:38.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.263+0000 7ff5dacd0700 1 -- 172.21.15.49:0/3531044023 shutdown_connections 2023-11-30T21:50:38.264 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.263+0000 7ff5dacd0700 1 -- 172.21.15.49:0/3531044023 wait complete. 2023-11-30T21:50:38.265 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.263+0000 7ff5dacd0700 1 Processor -- start 2023-11-30T21:50:38.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.263+0000 7ff5dacd0700 1 -- start start 2023-11-30T21:50:38.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.264+0000 7ff5dacd0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5d40f51a0 0x7ff5d41091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:38.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.264+0000 7ff5dacd0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d4060220 con 0x7ff5d40f51a0 2023-11-30T21:50:38.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.264+0000 7ff5d9cce700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5d40f51a0 0x7ff5d41091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:38.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.264+0000 7ff5d9cce700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5d40f51a0 0x7ff5d41091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:38714/0 (socket says 172.21.15.49:38714) 2023-11-30T21:50:38.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.264+0000 7ff5d9cce700 1 -- 172.21.15.49:0/1420216897 learned_addr learned my addr 172.21.15.49:0/1420216897 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:38.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.264+0000 7ff5d9cce700 1 -- 172.21.15.49:0/1420216897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d0005040 con 0x7ff5d40f51a0 2023-11-30T21:50:38.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.264+0000 7ff5d9cce700 1 --2- 172.21.15.49:0/1420216897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5d40f51a0 0x7ff5d41091d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7ff5d0013510 tx=0x7ff5d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:38.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.264+0000 7ff5ca7fc700 1 -- 172.21.15.49:0/1420216897 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5d0009320 con 0x7ff5d40f51a0 2023-11-30T21:50:38.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.265+0000 7ff5ca7fc700 1 -- 172.21.15.49:0/1420216897 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff5d0013f00 con 0x7ff5d40f51a0 2023-11-30T21:50:38.266 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.265+0000 7ff5dacd0700 1 -- 172.21.15.49:0/1420216897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d4060420 con 0x7ff5d40f51a0 2023-11-30T21:50:38.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.265+0000 7ff5ca7fc700 1 -- 172.21.15.49:0/1420216897 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5d0014dc0 con 0x7ff5d40f51a0 2023-11-30T21:50:38.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.265+0000 7ff5dacd0700 1 -- 172.21.15.49:0/1420216897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d4109880 con 0x7ff5d40f51a0 2023-11-30T21:50:38.267 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.265+0000 7ff5ca7fc700 1 -- 172.21.15.49:0/1420216897 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5d00095f0 con 0x7ff5d40f51a0 2023-11-30T21:50:38.268 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.265+0000 7ff5dacd0700 1 -- 172.21.15.49:0/1420216897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d4059100 con 0x7ff5d40f51a0 2023-11-30T21:50:38.268 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.266+0000 7ff5ca7fc700 1 --2- 172.21.15.49:0/1420216897 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff5c002ff60 0x7ff5c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:38.268 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.266+0000 7ff5ca7fc700 1 -- 172.21.15.49:0/1420216897 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff5d0041000 con 0x7ff5d40f51a0 2023-11-30T21:50:38.268 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.266+0000 7ff5d94cd700 1 --2- 172.21.15.49:0/1420216897 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff5c002ff60 0x7ff5c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:38.268 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.266+0000 7ff5d94cd700 1 --2- 172.21.15.49:0/1420216897 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff5c002ff60 0x7ff5c0032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7ff5c40096a0 tx=0x7ff5c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:38.271 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.270+0000 7ff5ca7fc700 1 -- 172.21.15.49:0/1420216897 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5d0014420 con 0x7ff5d40f51a0 2023-11-30T21:50:38.491 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.490+0000 7ff5dacd0700 1 -- 172.21.15.49:0/1420216897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5d4045ad0 con 0x7ff5d40f51a0 2023-11-30T21:50:38.492 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.491+0000 7ff5ca7fc700 1 -- 172.21.15.49:0/1420216897 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff5d0010230 con 0x7ff5d40f51a0 2023-11-30T21:50:38.492 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:38.493 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:38.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.492+0000 7ff5dacd0700 1 -- 172.21.15.49:0/1420216897 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff5c002ff60 msgr2=0x7ff5c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:38.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.492+0000 7ff5dacd0700 1 --2- 172.21.15.49:0/1420216897 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff5c002ff60 0x7ff5c0032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7ff5c40096a0 tx=0x7ff5c4006b40).stop 2023-11-30T21:50:38.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.492+0000 7ff5dacd0700 1 -- 172.21.15.49:0/1420216897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5d40f51a0 msgr2=0x7ff5d41091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:38.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.492+0000 7ff5dacd0700 1 --2- 172.21.15.49:0/1420216897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5d40f51a0 0x7ff5d41091d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7ff5d0013510 tx=0x7ff5d000b040).stop 2023-11-30T21:50:38.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.492+0000 7ff5dacd0700 1 -- 172.21.15.49:0/1420216897 shutdown_connections 2023-11-30T21:50:38.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.492+0000 7ff5dacd0700 1 --2- 172.21.15.49:0/1420216897 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff5c002ff60 0x7ff5c0032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:38.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.492+0000 7ff5dacd0700 1 --2- 172.21.15.49:0/1420216897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5d40f51a0 0x7ff5d41091d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:38.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.492+0000 7ff5dacd0700 1 -- 172.21.15.49:0/1420216897 >> 172.21.15.49:0/1420216897 conn(0x7ff5d40f09a0 msgr2=0x7ff5d40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:38.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.492+0000 7ff5dacd0700 1 -- 172.21.15.49:0/1420216897 shutdown_connections 2023-11-30T21:50:38.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:38.492+0000 7ff5dacd0700 1 -- 172.21.15.49:0/1420216897 wait complete. 2023-11-30T21:50:38.496 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:39.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:37.521216+0000 mgr.smithi027.brrjcl (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:39.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:38.491271+0000 mon.smithi027 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.49:0/1420216897' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:39.911 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:39.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:41.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.205+0000 7f3e0c26c700 1 -- 172.21.15.49:0/2868081665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e040f5410 msgr2=0x7f3e040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:41.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.205+0000 7f3e0c26c700 1 --2- 172.21.15.49:0/2868081665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e040f5410 0x7f3e040f5830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f3e00009fb0 tx=0x7f3e00009b30).stop 2023-11-30T21:50:41.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.205+0000 7f3e0c26c700 1 -- 172.21.15.49:0/2868081665 shutdown_connections 2023-11-30T21:50:41.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.205+0000 7f3e0c26c700 1 --2- 172.21.15.49:0/2868081665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e040f5410 0x7f3e040f5830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:41.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.205+0000 7f3e0c26c700 1 -- 172.21.15.49:0/2868081665 >> 172.21.15.49:0/2868081665 conn(0x7f3e040f0b70 msgr2=0x7f3e040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:41.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.205+0000 7f3e0c26c700 1 -- 172.21.15.49:0/2868081665 shutdown_connections 2023-11-30T21:50:41.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.205+0000 7f3e0c26c700 1 -- 172.21.15.49:0/2868081665 wait complete. 2023-11-30T21:50:41.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.206+0000 7f3e0c26c700 1 Processor -- start 2023-11-30T21:50:41.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.206+0000 7f3e0c26c700 1 -- start start 2023-11-30T21:50:41.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.206+0000 7f3e0c26c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e040f5410 0x7f3e04109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:41.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.206+0000 7f3e0c26c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e04060220 con 0x7f3e040f5410 2023-11-30T21:50:41.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.207+0000 7f3e0a008700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e040f5410 0x7f3e04109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:41.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.207+0000 7f3e0a008700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e040f5410 0x7f3e04109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:60706/0 (socket says 172.21.15.49:60706) 2023-11-30T21:50:41.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.207+0000 7f3e0a008700 1 -- 172.21.15.49:0/1066716308 learned_addr learned my addr 172.21.15.49:0/1066716308 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:41.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.207+0000 7f3e0a008700 1 -- 172.21.15.49:0/1066716308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e00005040 con 0x7f3e040f5410 2023-11-30T21:50:41.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.207+0000 7f3e0a008700 1 --2- 172.21.15.49:0/1066716308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e040f5410 0x7f3e04109450 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f3e00014000 tx=0x7f3e0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:41.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.207+0000 7f3df6ffd700 1 -- 172.21.15.49:0/1066716308 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e000095f0 con 0x7f3e040f5410 2023-11-30T21:50:41.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.207+0000 7f3df6ffd700 1 -- 172.21.15.49:0/1066716308 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e00007e70 con 0x7f3e040f5410 2023-11-30T21:50:41.209 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.207+0000 7f3e0c26c700 1 -- 172.21.15.49:0/1066716308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e04060420 con 0x7f3e040f5410 2023-11-30T21:50:41.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.207+0000 7f3df6ffd700 1 -- 172.21.15.49:0/1066716308 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e00014ea0 con 0x7f3e040f5410 2023-11-30T21:50:41.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.207+0000 7f3e0c26c700 1 -- 172.21.15.49:0/1066716308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e04109b00 con 0x7f3e040f5410 2023-11-30T21:50:41.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.208+0000 7f3df6ffd700 1 -- 172.21.15.49:0/1066716308 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3e00025070 con 0x7f3e040f5410 2023-11-30T21:50:41.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.208+0000 7f3e0c26c700 1 -- 172.21.15.49:0/1066716308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e04059100 con 0x7f3e040f5410 2023-11-30T21:50:41.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.208+0000 7f3df6ffd700 1 --2- 172.21.15.49:0/1066716308 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f3df002ff60 0x7f3df0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:41.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.208+0000 7f3df6ffd700 1 -- 172.21.15.49:0/1066716308 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e00040960 con 0x7f3e040f5410 2023-11-30T21:50:41.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.208+0000 7f3e09807700 1 --2- 172.21.15.49:0/1066716308 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f3df002ff60 0x7f3df0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:41.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.209+0000 7f3e09807700 1 --2- 172.21.15.49:0/1066716308 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f3df002ff60 0x7f3df0032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f3df80096a0 tx=0x7f3df8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:41.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.213+0000 7f3df6ffd700 1 -- 172.21.15.49:0/1066716308 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e000079c0 con 0x7f3e040f5410 2023-11-30T21:50:41.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.428+0000 7f3e0c26c700 1 -- 172.21.15.49:0/1066716308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e04045ad0 con 0x7f3e040f5410 2023-11-30T21:50:41.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.429+0000 7f3df6ffd700 1 -- 172.21.15.49:0/1066716308 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3e0000f300 con 0x7f3e040f5410 2023-11-30T21:50:41.430 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:41.430 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:41.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.430+0000 7f3e0c26c700 1 -- 172.21.15.49:0/1066716308 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f3df002ff60 msgr2=0x7f3df0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:41.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.430+0000 7f3e0c26c700 1 --2- 172.21.15.49:0/1066716308 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f3df002ff60 0x7f3df0032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f3df80096a0 tx=0x7f3df8006b40).stop 2023-11-30T21:50:41.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.430+0000 7f3e0c26c700 1 -- 172.21.15.49:0/1066716308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e040f5410 msgr2=0x7f3e04109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:41.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.430+0000 7f3e0c26c700 1 --2- 172.21.15.49:0/1066716308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e040f5410 0x7f3e04109450 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f3e00014000 tx=0x7f3e0000b040).stop 2023-11-30T21:50:41.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.430+0000 7f3e0c26c700 1 -- 172.21.15.49:0/1066716308 shutdown_connections 2023-11-30T21:50:41.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.430+0000 7f3e0c26c700 1 --2- 172.21.15.49:0/1066716308 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f3df002ff60 0x7f3df0032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:41.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.430+0000 7f3e0c26c700 1 --2- 172.21.15.49:0/1066716308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e040f5410 0x7f3e04109450 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:41.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.430+0000 7f3e0c26c700 1 -- 172.21.15.49:0/1066716308 >> 172.21.15.49:0/1066716308 conn(0x7f3e040f0b70 msgr2=0x7f3e040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:41.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.430+0000 7f3e0c26c700 1 -- 172.21.15.49:0/1066716308 shutdown_connections 2023-11-30T21:50:41.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:41.430+0000 7f3e0c26c700 1 -- 172.21.15.49:0/1066716308 wait complete. 2023-11-30T21:50:41.432 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:41.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:41 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:39.521476+0000 mgr.smithi027.brrjcl (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:42.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:42 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:41.429314+0000 mon.smithi027 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.49:0/1066716308' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:42.840 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:42.840 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:43.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:41.521812+0000 mgr.smithi027.brrjcl (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:44.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.139+0000 7f11031ca700 1 -- 172.21.15.49:0/2968533123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc0f5410 msgr2=0x7f10fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:44.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.139+0000 7f11031ca700 1 --2- 172.21.15.49:0/2968533123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc0f5410 0x7f10fc0f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f10f0009fb0 tx=0x7f10f0009b30).stop 2023-11-30T21:50:44.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.140+0000 7f11031ca700 1 -- 172.21.15.49:0/2968533123 shutdown_connections 2023-11-30T21:50:44.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.140+0000 7f11031ca700 1 --2- 172.21.15.49:0/2968533123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc0f5410 0x7f10fc0f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:44.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.140+0000 7f11031ca700 1 -- 172.21.15.49:0/2968533123 >> 172.21.15.49:0/2968533123 conn(0x7f10fc0f0b70 msgr2=0x7f10fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:44.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.140+0000 7f11031ca700 1 -- 172.21.15.49:0/2968533123 shutdown_connections 2023-11-30T21:50:44.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.140+0000 7f11031ca700 1 -- 172.21.15.49:0/2968533123 wait complete. 2023-11-30T21:50:44.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.140+0000 7f11031ca700 1 Processor -- start 2023-11-30T21:50:44.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.140+0000 7f11031ca700 1 -- start start 2023-11-30T21:50:44.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.140+0000 7f11031ca700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc0f5410 0x7f10fc109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:44.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.140+0000 7f11031ca700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc060220 con 0x7f10fc0f5410 2023-11-30T21:50:44.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.141+0000 7f1100f66700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc0f5410 0x7f10fc109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:44.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.141+0000 7f1100f66700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc0f5410 0x7f10fc109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:60724/0 (socket says 172.21.15.49:60724) 2023-11-30T21:50:44.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.141+0000 7f1100f66700 1 -- 172.21.15.49:0/1092913331 learned_addr learned my addr 172.21.15.49:0/1092913331 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:44.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.141+0000 7f1100f66700 1 -- 172.21.15.49:0/1092913331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f0005040 con 0x7f10fc0f5410 2023-11-30T21:50:44.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.141+0000 7f1100f66700 1 --2- 172.21.15.49:0/1092913331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc0f5410 0x7f10fc109450 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f10f0013510 tx=0x7f10f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:44.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.141+0000 7f10f97fa700 1 -- 172.21.15.49:0/1092913331 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10f0009320 con 0x7f10fc0f5410 2023-11-30T21:50:44.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.141+0000 7f10f97fa700 1 -- 172.21.15.49:0/1092913331 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f10f0013f00 con 0x7f10fc0f5410 2023-11-30T21:50:44.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.141+0000 7f11031ca700 1 -- 172.21.15.49:0/1092913331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10fc060420 con 0x7f10fc0f5410 2023-11-30T21:50:44.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.141+0000 7f10f97fa700 1 -- 172.21.15.49:0/1092913331 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10f0014d90 con 0x7f10fc0f5410 2023-11-30T21:50:44.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.142+0000 7f11031ca700 1 -- 172.21.15.49:0/1092913331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10fc109b00 con 0x7f10fc0f5410 2023-11-30T21:50:44.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.142+0000 7f10f97fa700 1 -- 172.21.15.49:0/1092913331 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10f0014420 con 0x7f10fc0f5410 2023-11-30T21:50:44.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.142+0000 7f11031ca700 1 -- 172.21.15.49:0/1092913331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10fc059100 con 0x7f10fc0f5410 2023-11-30T21:50:44.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.142+0000 7f10f97fa700 1 --2- 172.21.15.49:0/1092913331 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f10ec02ff60 0x7f10ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:44.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.142+0000 7f10f97fa700 1 -- 172.21.15.49:0/1092913331 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f10f00410c0 con 0x7f10fc0f5410 2023-11-30T21:50:44.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.143+0000 7f10fbfff700 1 --2- 172.21.15.49:0/1092913331 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f10ec02ff60 0x7f10ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:44.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.143+0000 7f10fbfff700 1 --2- 172.21.15.49:0/1092913331 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f10ec02ff60 0x7f10ec032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f10e80096a0 tx=0x7f10e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:44.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.147+0000 7f10f97fa700 1 -- 172.21.15.49:0/1092913331 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10f00095f0 con 0x7f10fc0f5410 2023-11-30T21:50:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.366+0000 7f11031ca700 1 -- 172.21.15.49:0/1092913331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10fc045ad0 con 0x7f10fc0f5410 2023-11-30T21:50:44.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.366+0000 7f10f97fa700 1 -- 172.21.15.49:0/1092913331 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f10f001db70 con 0x7f10fc0f5410 2023-11-30T21:50:44.369 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:44.369 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:44.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.368+0000 7f11031ca700 1 -- 172.21.15.49:0/1092913331 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f10ec02ff60 msgr2=0x7f10ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:44.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.368+0000 7f11031ca700 1 --2- 172.21.15.49:0/1092913331 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f10ec02ff60 0x7f10ec032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f10e80096a0 tx=0x7f10e8006b40).stop 2023-11-30T21:50:44.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.368+0000 7f11031ca700 1 -- 172.21.15.49:0/1092913331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc0f5410 msgr2=0x7f10fc109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:44.370 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.368+0000 7f11031ca700 1 --2- 172.21.15.49:0/1092913331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc0f5410 0x7f10fc109450 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f10f0013510 tx=0x7f10f000b040).stop 2023-11-30T21:50:44.371 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.368+0000 7f11031ca700 1 -- 172.21.15.49:0/1092913331 shutdown_connections 2023-11-30T21:50:44.371 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.368+0000 7f11031ca700 1 --2- 172.21.15.49:0/1092913331 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f10ec02ff60 0x7f10ec032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:44.371 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.368+0000 7f11031ca700 1 --2- 172.21.15.49:0/1092913331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10fc0f5410 0x7f10fc109450 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:44.371 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.368+0000 7f11031ca700 1 -- 172.21.15.49:0/1092913331 >> 172.21.15.49:0/1092913331 conn(0x7f10fc0f0b70 msgr2=0x7f10fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:44.371 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.368+0000 7f11031ca700 1 -- 172.21.15.49:0/1092913331 shutdown_connections 2023-11-30T21:50:44.372 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:44.368+0000 7f11031ca700 1 -- 172.21.15.49:0/1092913331 wait complete. 2023-11-30T21:50:44.372 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:45 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:43.522151+0000 mgr.smithi027.brrjcl (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:45 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:44.368113+0000 mon.smithi027 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.49:0/1092913331' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:45.767 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:45.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:47.076 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.073+0000 7efdd6614700 1 -- 172.21.15.49:0/3795075684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdd0060220 msgr2=0x7efdd00f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:47.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.073+0000 7efdd6614700 1 --2- 172.21.15.49:0/3795075684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdd0060220 0x7efdd00f9150 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7efdc0009fb0 tx=0x7efdc0009b30).stop 2023-11-30T21:50:47.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.073+0000 7efdd6614700 1 -- 172.21.15.49:0/3795075684 shutdown_connections 2023-11-30T21:50:47.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.073+0000 7efdd6614700 1 --2- 172.21.15.49:0/3795075684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdd0060220 0x7efdd00f9150 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:47.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.073+0000 7efdd6614700 1 -- 172.21.15.49:0/3795075684 >> 172.21.15.49:0/3795075684 conn(0x7efdd00f0b70 msgr2=0x7efdd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:47.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.073+0000 7efdd6614700 1 -- 172.21.15.49:0/3795075684 shutdown_connections 2023-11-30T21:50:47.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.073+0000 7efdd6614700 1 -- 172.21.15.49:0/3795075684 wait complete. 2023-11-30T21:50:47.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.073+0000 7efdd6614700 1 Processor -- start 2023-11-30T21:50:47.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.074+0000 7efdd6614700 1 -- start start 2023-11-30T21:50:47.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.074+0000 7efdd6614700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdd0060220 0x7efdd01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:47.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.074+0000 7efdd6614700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd0106bd0 con 0x7efdd0060220 2023-11-30T21:50:47.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.074+0000 7efdcffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdd0060220 0x7efdd01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:47.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.074+0000 7efdcffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdd0060220 0x7efdd01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:60746/0 (socket says 172.21.15.49:60746) 2023-11-30T21:50:47.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.074+0000 7efdcffff700 1 -- 172.21.15.49:0/1399509868 learned_addr learned my addr 172.21.15.49:0/1399509868 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:47.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.074+0000 7efdcffff700 1 -- 172.21.15.49:0/1399509868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc0005040 con 0x7efdd0060220 2023-11-30T21:50:47.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.074+0000 7efdcffff700 1 --2- 172.21.15.49:0/1399509868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdd0060220 0x7efdd01066e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7efdc0013510 tx=0x7efdc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:47.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.074+0000 7efdccff9700 1 -- 172.21.15.49:0/1399509868 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdc0009320 con 0x7efdd0060220 2023-11-30T21:50:47.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.075+0000 7efdd6614700 1 -- 172.21.15.49:0/1399509868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd0106dd0 con 0x7efdd0060220 2023-11-30T21:50:47.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.075+0000 7efdccff9700 1 -- 172.21.15.49:0/1399509868 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efdc0013f00 con 0x7efdd0060220 2023-11-30T21:50:47.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.075+0000 7efdccff9700 1 -- 172.21.15.49:0/1399509868 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efdc0014dc0 con 0x7efdd0060220 2023-11-30T21:50:47.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.075+0000 7efdd6614700 1 -- 172.21.15.49:0/1399509868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd0107260 con 0x7efdd0060220 2023-11-30T21:50:47.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.075+0000 7efdccff9700 1 -- 172.21.15.49:0/1399509868 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efdc00095f0 con 0x7efdd0060220 2023-11-30T21:50:47.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.075+0000 7efdd6614700 1 -- 172.21.15.49:0/1399509868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd0045ad0 con 0x7efdd0060220 2023-11-30T21:50:47.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.076+0000 7efdccff9700 1 --2- 172.21.15.49:0/1399509868 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efdb802ff60 0x7efdb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:47.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.076+0000 7efdccff9700 1 -- 172.21.15.49:0/1399509868 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efdc0041040 con 0x7efdd0060220 2023-11-30T21:50:47.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.076+0000 7efdcf7fe700 1 --2- 172.21.15.49:0/1399509868 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efdb802ff60 0x7efdb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:47.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.076+0000 7efdcf7fe700 1 --2- 172.21.15.49:0/1399509868 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efdb802ff60 0x7efdb8032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7efdc40096a0 tx=0x7efdc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:47.082 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.081+0000 7efdccff9700 1 -- 172.21.15.49:0/1399509868 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efdc0014420 con 0x7efdd0060220 2023-11-30T21:50:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.299+0000 7efdd6614700 1 -- 172.21.15.49:0/1399509868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efdd0108110 con 0x7efdd0060220 2023-11-30T21:50:47.301 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.300+0000 7efdccff9700 1 -- 172.21.15.49:0/1399509868 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efdc0010230 con 0x7efdd0060220 2023-11-30T21:50:47.301 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:47.302 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:47.303 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.301+0000 7efdd6614700 1 -- 172.21.15.49:0/1399509868 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efdb802ff60 msgr2=0x7efdb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:47.303 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.301+0000 7efdd6614700 1 --2- 172.21.15.49:0/1399509868 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efdb802ff60 0x7efdb8032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7efdc40096a0 tx=0x7efdc4006b40).stop 2023-11-30T21:50:47.303 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.301+0000 7efdd6614700 1 -- 172.21.15.49:0/1399509868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdd0060220 msgr2=0x7efdd01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:47.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.301+0000 7efdd6614700 1 --2- 172.21.15.49:0/1399509868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdd0060220 0x7efdd01066e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7efdc0013510 tx=0x7efdc000b040).stop 2023-11-30T21:50:47.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.301+0000 7efdd6614700 1 -- 172.21.15.49:0/1399509868 shutdown_connections 2023-11-30T21:50:47.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.301+0000 7efdd6614700 1 --2- 172.21.15.49:0/1399509868 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7efdb802ff60 0x7efdb8032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:47.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.301+0000 7efdd6614700 1 --2- 172.21.15.49:0/1399509868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdd0060220 0x7efdd01066e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:47.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.302+0000 7efdd6614700 1 -- 172.21.15.49:0/1399509868 >> 172.21.15.49:0/1399509868 conn(0x7efdd00f0b70 msgr2=0x7efdd00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:47.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.302+0000 7efdd6614700 1 -- 172.21.15.49:0/1399509868 shutdown_connections 2023-11-30T21:50:47.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:47.302+0000 7efdd6614700 1 -- 172.21.15.49:0/1399509868 wait complete. 2023-11-30T21:50:47.304 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:47.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:47 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:45.522434+0000 mgr.smithi027.brrjcl (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:48.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:48 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:47.301354+0000 mon.smithi027 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.49:0/1399509868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:48.705 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:48.705 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:49.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:49 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:47.522800+0000 mgr.smithi027.brrjcl (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.006+0000 7fba11af6700 1 -- 172.21.15.49:0/3101331878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba0c0f6510 msgr2=0x7fba0c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.006+0000 7fba11af6700 1 --2- 172.21.15.49:0/3101331878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba0c0f6510 0x7fba0c0f6930 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fb9fc009fb0 tx=0x7fb9fc009b30).stop 2023-11-30T21:50:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.006+0000 7fba11af6700 1 -- 172.21.15.49:0/3101331878 shutdown_connections 2023-11-30T21:50:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.006+0000 7fba11af6700 1 --2- 172.21.15.49:0/3101331878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba0c0f6510 0x7fba0c0f6930 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.006+0000 7fba11af6700 1 -- 172.21.15.49:0/3101331878 >> 172.21.15.49:0/3101331878 conn(0x7fba0c0f1c70 msgr2=0x7fba0c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.007+0000 7fba11af6700 1 -- 172.21.15.49:0/3101331878 shutdown_connections 2023-11-30T21:50:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.007+0000 7fba11af6700 1 -- 172.21.15.49:0/3101331878 wait complete. 2023-11-30T21:50:50.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.007+0000 7fba11af6700 1 Processor -- start 2023-11-30T21:50:50.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.007+0000 7fba11af6700 1 -- start start 2023-11-30T21:50:50.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.007+0000 7fba11af6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba0c0f6510 0x7fba0c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:50.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.007+0000 7fba11af6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c106c20 con 0x7fba0c0f6510 2023-11-30T21:50:50.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.008+0000 7fba0b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba0c0f6510 0x7fba0c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:50.011 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.008+0000 7fba0b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba0c0f6510 0x7fba0c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59920/0 (socket says 172.21.15.49:59920) 2023-11-30T21:50:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.008+0000 7fba0b7fe700 1 -- 172.21.15.49:0/1284751284 learned_addr learned my addr 172.21.15.49:0/1284751284 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.008+0000 7fba0b7fe700 1 -- 172.21.15.49:0/1284751284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc005040 con 0x7fba0c0f6510 2023-11-30T21:50:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.008+0000 7fba0b7fe700 1 --2- 172.21.15.49:0/1284751284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba0c0f6510 0x7fba0c106730 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fb9fc013510 tx=0x7fb9fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.008+0000 7fb9f3fff700 1 -- 172.21.15.49:0/1284751284 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9fc009320 con 0x7fba0c0f6510 2023-11-30T21:50:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.008+0000 7fb9f3fff700 1 -- 172.21.15.49:0/1284751284 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9fc013f00 con 0x7fba0c0f6510 2023-11-30T21:50:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.008+0000 7fba11af6700 1 -- 172.21.15.49:0/1284751284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0c106e20 con 0x7fba0c0f6510 2023-11-30T21:50:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.008+0000 7fb9f3fff700 1 -- 172.21.15.49:0/1284751284 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9fc014d90 con 0x7fba0c0f6510 2023-11-30T21:50:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.008+0000 7fba11af6700 1 -- 172.21.15.49:0/1284751284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0c1072b0 con 0x7fba0c0f6510 2023-11-30T21:50:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.009+0000 7fb9f3fff700 1 -- 172.21.15.49:0/1284751284 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9fc0263a0 con 0x7fba0c0f6510 2023-11-30T21:50:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.009+0000 7fba11af6700 1 -- 172.21.15.49:0/1284751284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0c045ad0 con 0x7fba0c0f6510 2023-11-30T21:50:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.009+0000 7fb9f3fff700 1 --2- 172.21.15.49:0/1284751284 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb9f402ff60 0x7fb9f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:50.012 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.009+0000 7fb9f3fff700 1 -- 172.21.15.49:0/1284751284 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9fc041300 con 0x7fba0c0f6510 2023-11-30T21:50:50.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.010+0000 7fba0affd700 1 --2- 172.21.15.49:0/1284751284 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb9f402ff60 0x7fb9f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:50.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.010+0000 7fba0affd700 1 --2- 172.21.15.49:0/1284751284 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb9f402ff60 0x7fb9f4032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fba000096a0 tx=0x7fba00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:50.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.014+0000 7fb9f3fff700 1 -- 172.21.15.49:0/1284751284 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9fc01cb10 con 0x7fba0c0f6510 2023-11-30T21:50:50.233 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.232+0000 7fba11af6700 1 -- 172.21.15.49:0/1284751284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba0c0fbf60 con 0x7fba0c0f6510 2023-11-30T21:50:50.234 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.232+0000 7fb9f3fff700 1 -- 172.21.15.49:0/1284751284 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb9fc01cb10 con 0x7fba0c0f6510 2023-11-30T21:50:50.234 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:50.234 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.234+0000 7fba11af6700 1 -- 172.21.15.49:0/1284751284 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb9f402ff60 msgr2=0x7fb9f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.234+0000 7fba11af6700 1 --2- 172.21.15.49:0/1284751284 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb9f402ff60 0x7fb9f4032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fba000096a0 tx=0x7fba00006b40).stop 2023-11-30T21:50:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.234+0000 7fba11af6700 1 -- 172.21.15.49:0/1284751284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba0c0f6510 msgr2=0x7fba0c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.234+0000 7fba11af6700 1 --2- 172.21.15.49:0/1284751284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba0c0f6510 0x7fba0c106730 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fb9fc013510 tx=0x7fb9fc00b040).stop 2023-11-30T21:50:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.234+0000 7fba11af6700 1 -- 172.21.15.49:0/1284751284 shutdown_connections 2023-11-30T21:50:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.234+0000 7fba11af6700 1 --2- 172.21.15.49:0/1284751284 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb9f402ff60 0x7fb9f4032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.234+0000 7fba11af6700 1 --2- 172.21.15.49:0/1284751284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba0c0f6510 0x7fba0c106730 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.234+0000 7fba11af6700 1 -- 172.21.15.49:0/1284751284 >> 172.21.15.49:0/1284751284 conn(0x7fba0c0f1c70 msgr2=0x7fba0c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:50.236 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.234+0000 7fba11af6700 1 -- 172.21.15.49:0/1284751284 shutdown_connections 2023-11-30T21:50:50.237 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:50.234+0000 7fba11af6700 1 -- 172.21.15.49:0/1284751284 wait complete. 2023-11-30T21:50:50.237 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:51.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:49.523169+0000 mgr.smithi027.brrjcl (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:51.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:50.233969+0000 mon.smithi027 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.49:0/1284751284' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:51.641 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:51.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:52.861 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.859+0000 7fc716216700 1 -- 172.21.15.49:0/1022317734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710060280 msgr2=0x7fc7100f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:52.861 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.859+0000 7fc716216700 1 --2- 172.21.15.49:0/1022317734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710060280 0x7fc7100f90f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fc704009fb0 tx=0x7fc704009b30).stop 2023-11-30T21:50:52.861 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.859+0000 7fc716216700 1 -- 172.21.15.49:0/1022317734 shutdown_connections 2023-11-30T21:50:52.861 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.859+0000 7fc716216700 1 --2- 172.21.15.49:0/1022317734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710060280 0x7fc7100f90f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:52.861 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.859+0000 7fc716216700 1 -- 172.21.15.49:0/1022317734 >> 172.21.15.49:0/1022317734 conn(0x7fc7100f0b70 msgr2=0x7fc7100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:52.861 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.859+0000 7fc716216700 1 -- 172.21.15.49:0/1022317734 shutdown_connections 2023-11-30T21:50:52.862 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.859+0000 7fc716216700 1 -- 172.21.15.49:0/1022317734 wait complete. 2023-11-30T21:50:52.862 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.860+0000 7fc716216700 1 Processor -- start 2023-11-30T21:50:52.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.860+0000 7fc716216700 1 -- start start 2023-11-30T21:50:52.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.860+0000 7fc716216700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710060280 0x7fc7101066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:52.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.860+0000 7fc716216700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc710106bd0 con 0x7fc710060280 2023-11-30T21:50:52.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.860+0000 7fc70f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710060280 0x7fc7101066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:52.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.860+0000 7fc70f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710060280 0x7fc7101066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59942/0 (socket says 172.21.15.49:59942) 2023-11-30T21:50:52.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.860+0000 7fc70f7fe700 1 -- 172.21.15.49:0/539306894 learned_addr learned my addr 172.21.15.49:0/539306894 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:52.863 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.861+0000 7fc70f7fe700 1 -- 172.21.15.49:0/539306894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc704005040 con 0x7fc710060280 2023-11-30T21:50:52.864 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.861+0000 7fc70f7fe700 1 --2- 172.21.15.49:0/539306894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710060280 0x7fc7101066e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fc704013510 tx=0x7fc70400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:52.864 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.861+0000 7fc6f7fff700 1 -- 172.21.15.49:0/539306894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc704009320 con 0x7fc710060280 2023-11-30T21:50:52.864 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.861+0000 7fc716216700 1 -- 172.21.15.49:0/539306894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc710106dd0 con 0x7fc710060280 2023-11-30T21:50:52.864 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.861+0000 7fc6f7fff700 1 -- 172.21.15.49:0/539306894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc704013f00 con 0x7fc710060280 2023-11-30T21:50:52.864 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.861+0000 7fc6f7fff700 1 -- 172.21.15.49:0/539306894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc704014dc0 con 0x7fc710060280 2023-11-30T21:50:52.864 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.861+0000 7fc716216700 1 -- 172.21.15.49:0/539306894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc710107260 con 0x7fc710060280 2023-11-30T21:50:52.865 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.862+0000 7fc6f7fff700 1 -- 172.21.15.49:0/539306894 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc70401d3d0 con 0x7fc710060280 2023-11-30T21:50:52.865 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.862+0000 7fc716216700 1 -- 172.21.15.49:0/539306894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc710045ad0 con 0x7fc710060280 2023-11-30T21:50:52.865 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.862+0000 7fc6f7fff700 1 --2- 172.21.15.49:0/539306894 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc6f802ff60 0x7fc6f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:52.865 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.862+0000 7fc6f7fff700 1 -- 172.21.15.49:0/539306894 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc704041030 con 0x7fc710060280 2023-11-30T21:50:52.865 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.862+0000 7fc70effd700 1 --2- 172.21.15.49:0/539306894 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc6f802ff60 0x7fc6f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:52.865 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.863+0000 7fc70effd700 1 --2- 172.21.15.49:0/539306894 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc6f802ff60 0x7fc6f8032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fc7000096a0 tx=0x7fc700006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:52.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:52.867+0000 7fc6f7fff700 1 -- 172.21.15.49:0/539306894 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7040095f0 con 0x7fc710060280 2023-11-30T21:50:53.086 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:53.084+0000 7fc716216700 1 -- 172.21.15.49:0/539306894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc710108110 con 0x7fc710060280 2023-11-30T21:50:53.087 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:53.085+0000 7fc6f7fff700 1 -- 172.21.15.49:0/539306894 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc70401c970 con 0x7fc710060280 2023-11-30T21:50:53.087 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:53.087 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:53.087+0000 7fc716216700 1 -- 172.21.15.49:0/539306894 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc6f802ff60 msgr2=0x7fc6f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:53.087+0000 7fc716216700 1 --2- 172.21.15.49:0/539306894 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc6f802ff60 0x7fc6f8032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fc7000096a0 tx=0x7fc700006b40).stop 2023-11-30T21:50:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:53.087+0000 7fc716216700 1 -- 172.21.15.49:0/539306894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710060280 msgr2=0x7fc7101066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:53.087+0000 7fc716216700 1 --2- 172.21.15.49:0/539306894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710060280 0x7fc7101066e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fc704013510 tx=0x7fc70400b040).stop 2023-11-30T21:50:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:53.087+0000 7fc716216700 1 -- 172.21.15.49:0/539306894 shutdown_connections 2023-11-30T21:50:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:53.087+0000 7fc716216700 1 --2- 172.21.15.49:0/539306894 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc6f802ff60 0x7fc6f8032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:53.087+0000 7fc716216700 1 --2- 172.21.15.49:0/539306894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc710060280 0x7fc7101066e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:53.087+0000 7fc716216700 1 -- 172.21.15.49:0/539306894 >> 172.21.15.49:0/539306894 conn(0x7fc7100f0b70 msgr2=0x7fc7100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:53.087+0000 7fc716216700 1 -- 172.21.15.49:0/539306894 shutdown_connections 2023-11-30T21:50:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:53.087+0000 7fc716216700 1 -- 172.21.15.49:0/539306894 wait complete. 2023-11-30T21:50:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:53.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:53 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:51.523396+0000 mgr.smithi027.brrjcl (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:53.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:53 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:53.086730+0000 mon.smithi027 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.49:0/539306894' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:54.465 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:54.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:55.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:53.523670+0000 mgr.smithi027.brrjcl (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:55.618 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.616+0000 7fb42fa99700 1 -- 172.21.15.49:0/2077071681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4280f5410 msgr2=0x7fb4280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:55.618 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.616+0000 7fb42fa99700 1 --2- 172.21.15.49:0/2077071681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4280f5410 0x7fb4280f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fb41c009fb0 tx=0x7fb41c009b30).stop 2023-11-30T21:50:55.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.616+0000 7fb42fa99700 1 -- 172.21.15.49:0/2077071681 shutdown_connections 2023-11-30T21:50:55.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.616+0000 7fb42fa99700 1 --2- 172.21.15.49:0/2077071681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4280f5410 0x7fb4280f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:55.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.616+0000 7fb42fa99700 1 -- 172.21.15.49:0/2077071681 >> 172.21.15.49:0/2077071681 conn(0x7fb4280f0b70 msgr2=0x7fb4280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:55.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.616+0000 7fb42fa99700 1 -- 172.21.15.49:0/2077071681 shutdown_connections 2023-11-30T21:50:55.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.616+0000 7fb42fa99700 1 -- 172.21.15.49:0/2077071681 wait complete. 2023-11-30T21:50:55.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.616+0000 7fb42fa99700 1 Processor -- start 2023-11-30T21:50:55.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.616+0000 7fb42fa99700 1 -- start start 2023-11-30T21:50:55.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.617+0000 7fb42fa99700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4280f5410 0x7fb428106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:55.619 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.617+0000 7fb42fa99700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb428106c10 con 0x7fb4280f5410 2023-11-30T21:50:55.620 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.617+0000 7fb42d835700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4280f5410 0x7fb428106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:55.620 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.617+0000 7fb42d835700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4280f5410 0x7fb428106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59974/0 (socket says 172.21.15.49:59974) 2023-11-30T21:50:55.620 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.617+0000 7fb42d835700 1 -- 172.21.15.49:0/355859019 learned_addr learned my addr 172.21.15.49:0/355859019 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:55.621 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.617+0000 7fb42d835700 1 -- 172.21.15.49:0/355859019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c005040 con 0x7fb4280f5410 2023-11-30T21:50:55.621 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.617+0000 7fb42d835700 1 --2- 172.21.15.49:0/355859019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4280f5410 0x7fb428106720 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fb41c014000 tx=0x7fb41c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:55.621 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.617+0000 7fb41a7fc700 1 -- 172.21.15.49:0/355859019 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb41c0095f0 con 0x7fb4280f5410 2023-11-30T21:50:55.621 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.617+0000 7fb42fa99700 1 -- 172.21.15.49:0/355859019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb428106e10 con 0x7fb4280f5410 2023-11-30T21:50:55.621 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.617+0000 7fb41a7fc700 1 -- 172.21.15.49:0/355859019 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb41c007e70 con 0x7fb4280f5410 2023-11-30T21:50:55.621 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.617+0000 7fb41a7fc700 1 -- 172.21.15.49:0/355859019 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb41c014ea0 con 0x7fb4280f5410 2023-11-30T21:50:55.622 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.618+0000 7fb42fa99700 1 -- 172.21.15.49:0/355859019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4281072a0 con 0x7fb4280f5410 2023-11-30T21:50:55.622 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.618+0000 7fb41a7fc700 1 -- 172.21.15.49:0/355859019 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb41c025070 con 0x7fb4280f5410 2023-11-30T21:50:55.622 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.618+0000 7fb41a7fc700 1 --2- 172.21.15.49:0/355859019 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb41402ff60 0x7fb414032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:55.622 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.618+0000 7fb42fa99700 1 -- 172.21.15.49:0/355859019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb428045ad0 con 0x7fb4280f5410 2023-11-30T21:50:55.622 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.618+0000 7fb41a7fc700 1 -- 172.21.15.49:0/355859019 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb41c040940 con 0x7fb4280f5410 2023-11-30T21:50:55.622 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.619+0000 7fb42d034700 1 --2- 172.21.15.49:0/355859019 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb41402ff60 0x7fb414032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:55.622 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.619+0000 7fb42d034700 1 --2- 172.21.15.49:0/355859019 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb41402ff60 0x7fb414032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fb4240096a0 tx=0x7fb424006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:55.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.625+0000 7fb41a7fc700 1 -- 172.21.15.49:0/355859019 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb41c0079c0 con 0x7fb4280f5410 2023-11-30T21:50:55.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.848+0000 7fb42fa99700 1 -- 172.21.15.49:0/355859019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4281080f0 con 0x7fb4280f5410 2023-11-30T21:50:55.850 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.849+0000 7fb41a7fc700 1 -- 172.21.15.49:0/355859019 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb41c02d090 con 0x7fb4280f5410 2023-11-30T21:50:55.851 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:55.851 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:55.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.850+0000 7fb42fa99700 1 -- 172.21.15.49:0/355859019 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb41402ff60 msgr2=0x7fb414032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:55.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.850+0000 7fb42fa99700 1 --2- 172.21.15.49:0/355859019 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb41402ff60 0x7fb414032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fb4240096a0 tx=0x7fb424006b40).stop 2023-11-30T21:50:55.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.850+0000 7fb42fa99700 1 -- 172.21.15.49:0/355859019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4280f5410 msgr2=0x7fb428106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:55.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.850+0000 7fb42fa99700 1 --2- 172.21.15.49:0/355859019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4280f5410 0x7fb428106720 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fb41c014000 tx=0x7fb41c00b040).stop 2023-11-30T21:50:55.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.850+0000 7fb42fa99700 1 -- 172.21.15.49:0/355859019 shutdown_connections 2023-11-30T21:50:55.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.850+0000 7fb42fa99700 1 --2- 172.21.15.49:0/355859019 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb41402ff60 0x7fb414032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:55.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.850+0000 7fb42fa99700 1 --2- 172.21.15.49:0/355859019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4280f5410 0x7fb428106720 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:55.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.850+0000 7fb42fa99700 1 -- 172.21.15.49:0/355859019 >> 172.21.15.49:0/355859019 conn(0x7fb4280f0b70 msgr2=0x7fb42805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:55.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.851+0000 7fb42fa99700 1 -- 172.21.15.49:0/355859019 shutdown_connections 2023-11-30T21:50:55.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:55.851+0000 7fb42fa99700 1 -- 172.21.15.49:0/355859019 wait complete. 2023-11-30T21:50:55.854 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:56.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:56 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:55.850216+0000 mon.smithi027 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.49:0/355859019' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:50:57.258 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:50:57.259 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:50:57.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:57 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:55.523948+0000 mgr.smithi027.brrjcl (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:58.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.572+0000 7feb99d53700 1 -- 172.21.15.49:0/1026476521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb940f51a0 msgr2=0x7feb940f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:58.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.572+0000 7feb99d53700 1 --2- 172.21.15.49:0/1026476521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb940f51a0 0x7feb940f55c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7feb88009fb0 tx=0x7feb88009b30).stop 2023-11-30T21:50:58.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.572+0000 7feb99d53700 1 -- 172.21.15.49:0/1026476521 shutdown_connections 2023-11-30T21:50:58.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.572+0000 7feb99d53700 1 --2- 172.21.15.49:0/1026476521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb940f51a0 0x7feb940f55c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:58.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.572+0000 7feb99d53700 1 -- 172.21.15.49:0/1026476521 >> 172.21.15.49:0/1026476521 conn(0x7feb940f09a0 msgr2=0x7feb940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:58.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.572+0000 7feb99d53700 1 -- 172.21.15.49:0/1026476521 shutdown_connections 2023-11-30T21:50:58.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.572+0000 7feb99d53700 1 -- 172.21.15.49:0/1026476521 wait complete. 2023-11-30T21:50:58.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.572+0000 7feb99d53700 1 Processor -- start 2023-11-30T21:50:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.572+0000 7feb99d53700 1 -- start start 2023-11-30T21:50:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.573+0000 7feb99d53700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb940f51a0 0x7feb941064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.573+0000 7feb99d53700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb94106990 con 0x7feb940f51a0 2023-11-30T21:50:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.573+0000 7feb98d51700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb940f51a0 0x7feb941064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.573+0000 7feb98d51700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb940f51a0 0x7feb941064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59376/0 (socket says 172.21.15.49:59376) 2023-11-30T21:50:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.573+0000 7feb98d51700 1 -- 172.21.15.49:0/1498489007 learned_addr learned my addr 172.21.15.49:0/1498489007 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:50:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.573+0000 7feb98d51700 1 -- 172.21.15.49:0/1498489007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb88005040 con 0x7feb940f51a0 2023-11-30T21:50:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.573+0000 7feb98d51700 1 --2- 172.21.15.49:0/1498489007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb940f51a0 0x7feb941064a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7feb88013510 tx=0x7feb8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.573+0000 7feb917fa700 1 -- 172.21.15.49:0/1498489007 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb88009320 con 0x7feb940f51a0 2023-11-30T21:50:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.574+0000 7feb99d53700 1 -- 172.21.15.49:0/1498489007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb94106b90 con 0x7feb940f51a0 2023-11-30T21:50:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.574+0000 7feb917fa700 1 -- 172.21.15.49:0/1498489007 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb88013f00 con 0x7feb940f51a0 2023-11-30T21:50:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.574+0000 7feb917fa700 1 -- 172.21.15.49:0/1498489007 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb88014dc0 con 0x7feb940f51a0 2023-11-30T21:50:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.574+0000 7feb99d53700 1 -- 172.21.15.49:0/1498489007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb94107020 con 0x7feb940f51a0 2023-11-30T21:50:58.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.574+0000 7feb917fa700 1 -- 172.21.15.49:0/1498489007 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb880095f0 con 0x7feb940f51a0 2023-11-30T21:50:58.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.574+0000 7feb99d53700 1 -- 172.21.15.49:0/1498489007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb94045ad0 con 0x7feb940f51a0 2023-11-30T21:50:58.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.575+0000 7feb917fa700 1 --2- 172.21.15.49:0/1498489007 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feb8402ff60 0x7feb84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:50:58.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.575+0000 7feb917fa700 1 -- 172.21.15.49:0/1498489007 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb88041040 con 0x7feb940f51a0 2023-11-30T21:50:58.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.575+0000 7feb93fff700 1 --2- 172.21.15.49:0/1498489007 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feb8402ff60 0x7feb84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:50:58.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.575+0000 7feb93fff700 1 --2- 172.21.15.49:0/1498489007 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feb8402ff60 0x7feb84032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7feb800096a0 tx=0x7feb80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:50:58.581 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.580+0000 7feb917fa700 1 -- 172.21.15.49:0/1498489007 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb88014420 con 0x7feb940f51a0 2023-11-30T21:50:58.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.799+0000 7feb99d53700 1 -- 172.21.15.49:0/1498489007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb94107ed0 con 0x7feb940f51a0 2023-11-30T21:50:58.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.800+0000 7feb917fa700 1 -- 172.21.15.49:0/1498489007 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb88010230 con 0x7feb940f51a0 2023-11-30T21:50:58.801 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:50:58.802 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:50:58.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.801+0000 7feb99d53700 1 -- 172.21.15.49:0/1498489007 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feb8402ff60 msgr2=0x7feb84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:58.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.801+0000 7feb99d53700 1 --2- 172.21.15.49:0/1498489007 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feb8402ff60 0x7feb84032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7feb800096a0 tx=0x7feb80006b40).stop 2023-11-30T21:50:58.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.801+0000 7feb99d53700 1 -- 172.21.15.49:0/1498489007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb940f51a0 msgr2=0x7feb941064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:50:58.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.801+0000 7feb99d53700 1 --2- 172.21.15.49:0/1498489007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb940f51a0 0x7feb941064a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7feb88013510 tx=0x7feb8800b040).stop 2023-11-30T21:50:58.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.801+0000 7feb99d53700 1 -- 172.21.15.49:0/1498489007 shutdown_connections 2023-11-30T21:50:58.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.801+0000 7feb99d53700 1 --2- 172.21.15.49:0/1498489007 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feb8402ff60 0x7feb84032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:58.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.801+0000 7feb99d53700 1 --2- 172.21.15.49:0/1498489007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb940f51a0 0x7feb941064a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:50:58.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.801+0000 7feb99d53700 1 -- 172.21.15.49:0/1498489007 >> 172.21.15.49:0/1498489007 conn(0x7feb940f09a0 msgr2=0x7feb9405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:50:58.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.801+0000 7feb99d53700 1 -- 172.21.15.49:0/1498489007 shutdown_connections 2023-11-30T21:50:58.804 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:50:58.801+0000 7feb99d53700 1 -- 172.21.15.49:0/1498489007 wait complete. 2023-11-30T21:50:58.804 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:50:59.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:59 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:57.524290+0000 mgr.smithi027.brrjcl (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:50:59.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:50:59 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:50:58.801232+0000 mon.smithi027 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.49:0/1498489007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:00.208 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:00.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:01.479 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.473+0000 7fadb759e700 1 -- 172.21.15.49:0/3328006927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80f51a0 msgr2=0x7fadb80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:01.479 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.473+0000 7fadb759e700 1 --2- 172.21.15.49:0/3328006927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80f51a0 0x7fadb80f55c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fada8009fb0 tx=0x7fada8009b30).stop 2023-11-30T21:51:01.479 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.473+0000 7fadb759e700 1 -- 172.21.15.49:0/3328006927 shutdown_connections 2023-11-30T21:51:01.479 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.473+0000 7fadb759e700 1 --2- 172.21.15.49:0/3328006927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80f51a0 0x7fadb80f55c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:01.480 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.473+0000 7fadb759e700 1 -- 172.21.15.49:0/3328006927 >> 172.21.15.49:0/3328006927 conn(0x7fadb80f09a0 msgr2=0x7fadb80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:01.480 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.473+0000 7fadb759e700 1 -- 172.21.15.49:0/3328006927 shutdown_connections 2023-11-30T21:51:01.480 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.473+0000 7fadb759e700 1 -- 172.21.15.49:0/3328006927 wait complete. 2023-11-30T21:51:01.480 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.473+0000 7fadb759e700 1 Processor -- start 2023-11-30T21:51:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.474+0000 7fadb759e700 1 -- start start 2023-11-30T21:51:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.474+0000 7fadb759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80f51a0 0x7fadb81064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.474+0000 7fadb759e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb8106990 con 0x7fadb80f51a0 2023-11-30T21:51:01.481 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.474+0000 7fadb659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80f51a0 0x7fadb81064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:01.482 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.474+0000 7fadb659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80f51a0 0x7fadb81064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59400/0 (socket says 172.21.15.49:59400) 2023-11-30T21:51:01.482 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.474+0000 7fadb659c700 1 -- 172.21.15.49:0/2166934680 learned_addr learned my addr 172.21.15.49:0/2166934680 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:01.482 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.474+0000 7fadb659c700 1 -- 172.21.15.49:0/2166934680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada8005040 con 0x7fadb80f51a0 2023-11-30T21:51:01.482 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.474+0000 7fadb659c700 1 --2- 172.21.15.49:0/2166934680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80f51a0 0x7fadb81064a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fada8013510 tx=0x7fada800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:01.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.475+0000 7fadaeffd700 1 -- 172.21.15.49:0/2166934680 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fada8009320 con 0x7fadb80f51a0 2023-11-30T21:51:01.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.475+0000 7fadb759e700 1 -- 172.21.15.49:0/2166934680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb8106b90 con 0x7fadb80f51a0 2023-11-30T21:51:01.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.475+0000 7fadaeffd700 1 -- 172.21.15.49:0/2166934680 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fada8013f00 con 0x7fadb80f51a0 2023-11-30T21:51:01.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.475+0000 7fadaeffd700 1 -- 172.21.15.49:0/2166934680 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fada8014dc0 con 0x7fadb80f51a0 2023-11-30T21:51:01.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.475+0000 7fadb759e700 1 -- 172.21.15.49:0/2166934680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb8107020 con 0x7fadb80f51a0 2023-11-30T21:51:01.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.476+0000 7fadaeffd700 1 -- 172.21.15.49:0/2166934680 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fada80095f0 con 0x7fadb80f51a0 2023-11-30T21:51:01.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.476+0000 7fadb759e700 1 -- 172.21.15.49:0/2166934680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb8045ad0 con 0x7fadb80f51a0 2023-11-30T21:51:01.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.476+0000 7fadaeffd700 1 --2- 172.21.15.49:0/2166934680 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fada402ff60 0x7fada4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:01.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.476+0000 7fadaeffd700 1 -- 172.21.15.49:0/2166934680 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fada8041000 con 0x7fadb80f51a0 2023-11-30T21:51:01.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.476+0000 7fadb5d9b700 1 --2- 172.21.15.49:0/2166934680 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fada402ff60 0x7fada4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:01.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.476+0000 7fadb5d9b700 1 --2- 172.21.15.49:0/2166934680 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fada402ff60 0x7fada4032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fada00096a0 tx=0x7fada0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:01.486 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.481+0000 7fadaeffd700 1 -- 172.21.15.49:0/2166934680 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fada801c790 con 0x7fadb80f51a0 2023-11-30T21:51:01.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:01 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:50:59.524655+0000 mgr.smithi027.brrjcl (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:01.700 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.699+0000 7fadb759e700 1 -- 172.21.15.49:0/2166934680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fadb8107ed0 con 0x7fadb80f51a0 2023-11-30T21:51:01.701 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.699+0000 7fadaeffd700 1 -- 172.21.15.49:0/2166934680 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fada8010230 con 0x7fadb80f51a0 2023-11-30T21:51:01.701 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:01.701 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:01.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.701+0000 7fadb759e700 1 -- 172.21.15.49:0/2166934680 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fada402ff60 msgr2=0x7fada4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:01.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.701+0000 7fadb759e700 1 --2- 172.21.15.49:0/2166934680 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fada402ff60 0x7fada4032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fada00096a0 tx=0x7fada0006b40).stop 2023-11-30T21:51:01.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.701+0000 7fadb759e700 1 -- 172.21.15.49:0/2166934680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80f51a0 msgr2=0x7fadb81064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:01.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.701+0000 7fadb759e700 1 --2- 172.21.15.49:0/2166934680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80f51a0 0x7fadb81064a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fada8013510 tx=0x7fada800b040).stop 2023-11-30T21:51:01.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.701+0000 7fadb759e700 1 -- 172.21.15.49:0/2166934680 shutdown_connections 2023-11-30T21:51:01.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.701+0000 7fadb759e700 1 --2- 172.21.15.49:0/2166934680 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fada402ff60 0x7fada4032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:01.703 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.701+0000 7fadb759e700 1 --2- 172.21.15.49:0/2166934680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80f51a0 0x7fadb81064a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:01.704 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.701+0000 7fadb759e700 1 -- 172.21.15.49:0/2166934680 >> 172.21.15.49:0/2166934680 conn(0x7fadb80f09a0 msgr2=0x7fadb805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:01.704 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.701+0000 7fadb759e700 1 -- 172.21.15.49:0/2166934680 shutdown_connections 2023-11-30T21:51:01.704 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:01.701+0000 7fadb759e700 1 -- 172.21.15.49:0/2166934680 wait complete. 2023-11-30T21:51:01.704 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:02.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:02 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:01.700936+0000 mon.smithi027 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.49:0/2166934680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:03.134 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:03.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:03.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:03 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:01.524967+0000 mgr.smithi027.brrjcl (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:04.381 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.380+0000 7f169345b700 1 -- 172.21.15.49:0/277966337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0f62a0 msgr2=0x7f168c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:04.382 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.380+0000 7f169345b700 1 --2- 172.21.15.49:0/277966337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0f62a0 0x7f168c0f66c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f167c009fb0 tx=0x7f167c009b30).stop 2023-11-30T21:51:04.382 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.380+0000 7f169345b700 1 -- 172.21.15.49:0/277966337 shutdown_connections 2023-11-30T21:51:04.382 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.380+0000 7f169345b700 1 --2- 172.21.15.49:0/277966337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0f62a0 0x7f168c0f66c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:04.382 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.380+0000 7f169345b700 1 -- 172.21.15.49:0/277966337 >> 172.21.15.49:0/277966337 conn(0x7f168c0f1aa0 msgr2=0x7f168c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:04.382 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.380+0000 7f169345b700 1 -- 172.21.15.49:0/277966337 shutdown_connections 2023-11-30T21:51:04.383 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.380+0000 7f169345b700 1 -- 172.21.15.49:0/277966337 wait complete. 2023-11-30T21:51:04.385 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.380+0000 7f169345b700 1 Processor -- start 2023-11-30T21:51:04.385 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.380+0000 7f169345b700 1 -- start start 2023-11-30T21:51:04.385 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.381+0000 7f169345b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0f62a0 0x7f168c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:04.386 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.381+0000 7f169345b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c1069a0 con 0x7f168c0f62a0 2023-11-30T21:51:04.386 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.381+0000 7f1692459700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0f62a0 0x7f168c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:04.387 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.381+0000 7f1692459700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0f62a0 0x7f168c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59418/0 (socket says 172.21.15.49:59418) 2023-11-30T21:51:04.387 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.381+0000 7f1692459700 1 -- 172.21.15.49:0/203208331 learned_addr learned my addr 172.21.15.49:0/203208331 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:04.387 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.381+0000 7f1692459700 1 -- 172.21.15.49:0/203208331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c005040 con 0x7f168c0f62a0 2023-11-30T21:51:04.387 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.381+0000 7f1692459700 1 --2- 172.21.15.49:0/203208331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0f62a0 0x7f168c1064b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f167c013510 tx=0x7f167c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:04.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.382+0000 7f1682ffd700 1 -- 172.21.15.49:0/203208331 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167c009320 con 0x7f168c0f62a0 2023-11-30T21:51:04.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.382+0000 7f169345b700 1 -- 172.21.15.49:0/203208331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168c106ba0 con 0x7f168c0f62a0 2023-11-30T21:51:04.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.382+0000 7f1682ffd700 1 -- 172.21.15.49:0/203208331 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f167c013f00 con 0x7f168c0f62a0 2023-11-30T21:51:04.388 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.382+0000 7f1682ffd700 1 -- 172.21.15.49:0/203208331 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167c014dc0 con 0x7f168c0f62a0 2023-11-30T21:51:04.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.382+0000 7f169345b700 1 -- 172.21.15.49:0/203208331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168c107030 con 0x7f168c0f62a0 2023-11-30T21:51:04.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.382+0000 7f1682ffd700 1 -- 172.21.15.49:0/203208331 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f167c0095f0 con 0x7f168c0f62a0 2023-11-30T21:51:04.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.382+0000 7f169345b700 1 -- 172.21.15.49:0/203208331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168c045ad0 con 0x7f168c0f62a0 2023-11-30T21:51:04.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.383+0000 7f1682ffd700 1 --2- 172.21.15.49:0/203208331 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f167802ff60 0x7f1678032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:04.389 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.383+0000 7f1682ffd700 1 -- 172.21.15.49:0/203208331 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f167c041040 con 0x7f168c0f62a0 2023-11-30T21:51:04.390 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.383+0000 7f1691c58700 1 --2- 172.21.15.49:0/203208331 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f167802ff60 0x7f1678032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:04.390 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.383+0000 7f1691c58700 1 --2- 172.21.15.49:0/203208331 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f167802ff60 0x7f1678032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f16880096a0 tx=0x7f1688006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:04.391 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.387+0000 7f1682ffd700 1 -- 172.21.15.49:0/203208331 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f167c014420 con 0x7f168c0f62a0 2023-11-30T21:51:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:03.591219+0000 mon.smithi027 (mon.0) 269 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/mirror_snapshot_schedule"}]: dispatch 2023-11-30T21:51:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:03.597069+0000 mon.smithi027 (mon.0) 270 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/trash_purge_schedule"}]: dispatch 2023-11-30T21:51:04.607 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.605+0000 7f169345b700 1 -- 172.21.15.49:0/203208331 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f168c107ee0 con 0x7f168c0f62a0 2023-11-30T21:51:04.608 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.606+0000 7f1682ffd700 1 -- 172.21.15.49:0/203208331 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f167c010230 con 0x7f168c0f62a0 2023-11-30T21:51:04.608 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:04.608 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:04.610 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.608+0000 7f169345b700 1 -- 172.21.15.49:0/203208331 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f167802ff60 msgr2=0x7f1678032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:04.611 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.608+0000 7f169345b700 1 --2- 172.21.15.49:0/203208331 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f167802ff60 0x7f1678032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f16880096a0 tx=0x7f1688006b40).stop 2023-11-30T21:51:04.611 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.608+0000 7f169345b700 1 -- 172.21.15.49:0/203208331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0f62a0 msgr2=0x7f168c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:04.611 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.608+0000 7f169345b700 1 --2- 172.21.15.49:0/203208331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0f62a0 0x7f168c1064b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f167c013510 tx=0x7f167c00b040).stop 2023-11-30T21:51:04.611 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.608+0000 7f169345b700 1 -- 172.21.15.49:0/203208331 shutdown_connections 2023-11-30T21:51:04.611 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.608+0000 7f169345b700 1 --2- 172.21.15.49:0/203208331 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f167802ff60 0x7f1678032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:04.611 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.608+0000 7f169345b700 1 --2- 172.21.15.49:0/203208331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0f62a0 0x7f168c1064b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:04.612 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.608+0000 7f169345b700 1 -- 172.21.15.49:0/203208331 >> 172.21.15.49:0/203208331 conn(0x7f168c0f1aa0 msgr2=0x7f168c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:04.612 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.608+0000 7f169345b700 1 -- 172.21.15.49:0/203208331 shutdown_connections 2023-11-30T21:51:04.612 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:04.608+0000 7f169345b700 1 -- 172.21.15.49:0/203208331 wait complete. 2023-11-30T21:51:04.612 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:05.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:03.525287+0000 mgr.smithi027.brrjcl (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:05.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:04.607715+0000 mon.smithi027 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.49:0/203208331' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:05.936 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:05.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:07.189 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.188+0000 7fc615449700 1 -- 172.21.15.49:0/1107995988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6100f6510 msgr2=0x7fc6100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:07.189 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.188+0000 7fc615449700 1 --2- 172.21.15.49:0/1107995988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6100f6510 0x7fc6100f6930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fc600009fb0 tx=0x7fc600009b30).stop 2023-11-30T21:51:07.189 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.188+0000 7fc615449700 1 -- 172.21.15.49:0/1107995988 shutdown_connections 2023-11-30T21:51:07.189 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.188+0000 7fc615449700 1 --2- 172.21.15.49:0/1107995988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6100f6510 0x7fc6100f6930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:07.189 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.188+0000 7fc615449700 1 -- 172.21.15.49:0/1107995988 >> 172.21.15.49:0/1107995988 conn(0x7fc6100f1c70 msgr2=0x7fc6100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:07.189 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.188+0000 7fc615449700 1 -- 172.21.15.49:0/1107995988 shutdown_connections 2023-11-30T21:51:07.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.188+0000 7fc615449700 1 -- 172.21.15.49:0/1107995988 wait complete. 2023-11-30T21:51:07.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.188+0000 7fc615449700 1 Processor -- start 2023-11-30T21:51:07.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.188+0000 7fc615449700 1 -- start start 2023-11-30T21:51:07.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.189+0000 7fc615449700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6100f6510 0x7fc6101088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:07.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.189+0000 7fc615449700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc610108dd0 con 0x7fc6100f6510 2023-11-30T21:51:07.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.189+0000 7fc60effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6100f6510 0x7fc6101088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:07.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.189+0000 7fc60effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6100f6510 0x7fc6101088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59448/0 (socket says 172.21.15.49:59448) 2023-11-30T21:51:07.192 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.189+0000 7fc60effd700 1 -- 172.21.15.49:0/2559554896 learned_addr learned my addr 172.21.15.49:0/2559554896 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:07.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.189+0000 7fc60effd700 1 -- 172.21.15.49:0/2559554896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc600005040 con 0x7fc6100f6510 2023-11-30T21:51:07.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.189+0000 7fc60effd700 1 --2- 172.21.15.49:0/2559554896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6100f6510 0x7fc6101088e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fc600013510 tx=0x7fc60000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:07.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.189+0000 7fc6077fe700 1 -- 172.21.15.49:0/2559554896 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc600009320 con 0x7fc6100f6510 2023-11-30T21:51:07.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.190+0000 7fc6077fe700 1 -- 172.21.15.49:0/2559554896 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc600013f00 con 0x7fc6100f6510 2023-11-30T21:51:07.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.190+0000 7fc615449700 1 -- 172.21.15.49:0/2559554896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc610108fd0 con 0x7fc6100f6510 2023-11-30T21:51:07.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.190+0000 7fc6077fe700 1 -- 172.21.15.49:0/2559554896 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc600014d90 con 0x7fc6100f6510 2023-11-30T21:51:07.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.190+0000 7fc615449700 1 -- 172.21.15.49:0/2559554896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc610109460 con 0x7fc6100f6510 2023-11-30T21:51:07.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.190+0000 7fc6077fe700 1 -- 172.21.15.49:0/2559554896 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6000263a0 con 0x7fc6100f6510 2023-11-30T21:51:07.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.190+0000 7fc615449700 1 -- 172.21.15.49:0/2559554896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc610045ad0 con 0x7fc6100f6510 2023-11-30T21:51:07.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.191+0000 7fc6077fe700 1 --2- 172.21.15.49:0/2559554896 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc5fc02ffb0 0x7fc5fc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:07.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.191+0000 7fc6077fe700 1 -- 172.21.15.49:0/2559554896 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc6000412a0 con 0x7fc6100f6510 2023-11-30T21:51:07.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.191+0000 7fc60e7fc700 1 --2- 172.21.15.49:0/2559554896 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc5fc02ffb0 0x7fc5fc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:07.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.191+0000 7fc60e7fc700 1 --2- 172.21.15.49:0/2559554896 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc5fc02ffb0 0x7fc5fc032470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fc5f80096a0 tx=0x7fc5f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:07.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.195+0000 7fc6077fe700 1 -- 172.21.15.49:0/2559554896 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc60001cb10 con 0x7fc6100f6510 2023-11-30T21:51:07.417 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.416+0000 7fc615449700 1 -- 172.21.15.49:0/2559554896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc61006b280 con 0x7fc6100f6510 2023-11-30T21:51:07.418 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.417+0000 7fc6077fe700 1 -- 172.21.15.49:0/2559554896 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc60001cb10 con 0x7fc6100f6510 2023-11-30T21:51:07.418 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:07.418 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:07.419 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.418+0000 7fc615449700 1 -- 172.21.15.49:0/2559554896 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc5fc02ffb0 msgr2=0x7fc5fc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:07.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.418+0000 7fc615449700 1 --2- 172.21.15.49:0/2559554896 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc5fc02ffb0 0x7fc5fc032470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fc5f80096a0 tx=0x7fc5f8006b40).stop 2023-11-30T21:51:07.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.418+0000 7fc615449700 1 -- 172.21.15.49:0/2559554896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6100f6510 msgr2=0x7fc6101088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:07.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.418+0000 7fc615449700 1 --2- 172.21.15.49:0/2559554896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6100f6510 0x7fc6101088e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fc600013510 tx=0x7fc60000b040).stop 2023-11-30T21:51:07.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.418+0000 7fc615449700 1 -- 172.21.15.49:0/2559554896 shutdown_connections 2023-11-30T21:51:07.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.418+0000 7fc615449700 1 --2- 172.21.15.49:0/2559554896 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc5fc02ffb0 0x7fc5fc032470 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:07.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.418+0000 7fc615449700 1 --2- 172.21.15.49:0/2559554896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6100f6510 0x7fc6101088e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:07.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.418+0000 7fc615449700 1 -- 172.21.15.49:0/2559554896 >> 172.21.15.49:0/2559554896 conn(0x7fc6100f1c70 msgr2=0x7fc6100f4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:07.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.418+0000 7fc615449700 1 -- 172.21.15.49:0/2559554896 shutdown_connections 2023-11-30T21:51:07.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:07.418+0000 7fc615449700 1 -- 172.21.15.49:0/2559554896 wait complete. 2023-11-30T21:51:07.421 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:07.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:07 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:05.525610+0000 mgr.smithi027.brrjcl (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:08.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:08 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:07.417253+0000 mon.smithi027 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.49:0/2559554896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:08.876 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:08.877 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:09.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:09 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:07.526003+0000 mgr.smithi027.brrjcl (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:10.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.138+0000 7f5b6525b700 1 -- 172.21.15.49:0/2882940215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b600f5410 msgr2=0x7f5b600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:10.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.138+0000 7f5b6525b700 1 --2- 172.21.15.49:0/2882940215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b600f5410 0x7f5b600f5830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f5b54009fb0 tx=0x7f5b54009b30).stop 2023-11-30T21:51:10.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.138+0000 7f5b6525b700 1 -- 172.21.15.49:0/2882940215 shutdown_connections 2023-11-30T21:51:10.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.138+0000 7f5b6525b700 1 --2- 172.21.15.49:0/2882940215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b600f5410 0x7f5b600f5830 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:10.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.138+0000 7f5b6525b700 1 -- 172.21.15.49:0/2882940215 >> 172.21.15.49:0/2882940215 conn(0x7f5b600f0b70 msgr2=0x7f5b600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:10.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.138+0000 7f5b6525b700 1 -- 172.21.15.49:0/2882940215 shutdown_connections 2023-11-30T21:51:10.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.138+0000 7f5b6525b700 1 -- 172.21.15.49:0/2882940215 wait complete. 2023-11-30T21:51:10.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.139+0000 7f5b6525b700 1 Processor -- start 2023-11-30T21:51:10.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.139+0000 7f5b6525b700 1 -- start start 2023-11-30T21:51:10.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.139+0000 7f5b6525b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b600f5410 0x7f5b600fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:10.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.139+0000 7f5b6525b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b600fb820 con 0x7f5b600f5410 2023-11-30T21:51:10.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.139+0000 7f5b5ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b600f5410 0x7f5b600fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:10.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.139+0000 7f5b5ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b600f5410 0x7f5b600fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:58470/0 (socket says 172.21.15.49:58470) 2023-11-30T21:51:10.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.139+0000 7f5b5ed9d700 1 -- 172.21.15.49:0/3725575131 learned_addr learned my addr 172.21.15.49:0/3725575131 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:10.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.139+0000 7f5b5ed9d700 1 -- 172.21.15.49:0/3725575131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b54005040 con 0x7f5b600f5410 2023-11-30T21:51:10.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.139+0000 7f5b5ed9d700 1 --2- 172.21.15.49:0/3725575131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b600f5410 0x7f5b600fd3a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f5b54013510 tx=0x7f5b5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:10.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.140+0000 7f5b477fe700 1 -- 172.21.15.49:0/3725575131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b54009320 con 0x7f5b600f5410 2023-11-30T21:51:10.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.140+0000 7f5b6525b700 1 -- 172.21.15.49:0/3725575131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b600fb9c0 con 0x7f5b600f5410 2023-11-30T21:51:10.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.140+0000 7f5b477fe700 1 -- 172.21.15.49:0/3725575131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b54013f00 con 0x7f5b600f5410 2023-11-30T21:51:10.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.140+0000 7f5b477fe700 1 -- 172.21.15.49:0/3725575131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b54014d90 con 0x7f5b600f5410 2023-11-30T21:51:10.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.140+0000 7f5b6525b700 1 -- 172.21.15.49:0/3725575131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b600fbe50 con 0x7f5b600f5410 2023-11-30T21:51:10.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.140+0000 7f5b6525b700 1 -- 172.21.15.49:0/3725575131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b60059100 con 0x7f5b600f5410 2023-11-30T21:51:10.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.141+0000 7f5b477fe700 1 -- 172.21.15.49:0/3725575131 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b540095f0 con 0x7f5b600f5410 2023-11-30T21:51:10.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.141+0000 7f5b477fe700 1 --2- 172.21.15.49:0/3725575131 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5b4802ff60 0x7f5b48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:10.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.141+0000 7f5b477fe700 1 -- 172.21.15.49:0/3725575131 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b54041000 con 0x7f5b600f5410 2023-11-30T21:51:10.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.141+0000 7f5b5e59c700 1 --2- 172.21.15.49:0/3725575131 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5b4802ff60 0x7f5b48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:10.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.141+0000 7f5b5e59c700 1 --2- 172.21.15.49:0/3725575131 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5b4802ff60 0x7f5b48032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f5b500096a0 tx=0x7f5b50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:10.147 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.146+0000 7f5b477fe700 1 -- 172.21.15.49:0/3725575131 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b54014420 con 0x7f5b600f5410 2023-11-30T21:51:10.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.362+0000 7f5b6525b700 1 -- 172.21.15.49:0/3725575131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b600fae60 con 0x7f5b600f5410 2023-11-30T21:51:10.364 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.363+0000 7f5b477fe700 1 -- 172.21.15.49:0/3725575131 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5b54010230 con 0x7f5b600f5410 2023-11-30T21:51:10.365 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:10.365 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.364+0000 7f5b6525b700 1 -- 172.21.15.49:0/3725575131 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5b4802ff60 msgr2=0x7f5b48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.364+0000 7f5b6525b700 1 --2- 172.21.15.49:0/3725575131 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5b4802ff60 0x7f5b48032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f5b500096a0 tx=0x7f5b50006b40).stop 2023-11-30T21:51:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.364+0000 7f5b6525b700 1 -- 172.21.15.49:0/3725575131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b600f5410 msgr2=0x7f5b600fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.365+0000 7f5b6525b700 1 --2- 172.21.15.49:0/3725575131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b600f5410 0x7f5b600fd3a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f5b54013510 tx=0x7f5b5400b040).stop 2023-11-30T21:51:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.365+0000 7f5b6525b700 1 -- 172.21.15.49:0/3725575131 shutdown_connections 2023-11-30T21:51:10.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.365+0000 7f5b6525b700 1 --2- 172.21.15.49:0/3725575131 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5b4802ff60 0x7f5b48032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:10.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.365+0000 7f5b6525b700 1 --2- 172.21.15.49:0/3725575131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b600f5410 0x7f5b600fd3a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:10.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.365+0000 7f5b6525b700 1 -- 172.21.15.49:0/3725575131 >> 172.21.15.49:0/3725575131 conn(0x7f5b600f0b70 msgr2=0x7f5b6005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:10.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.365+0000 7f5b6525b700 1 -- 172.21.15.49:0/3725575131 shutdown_connections 2023-11-30T21:51:10.368 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:10.365+0000 7f5b6525b700 1 -- 172.21.15.49:0/3725575131 wait complete. 2023-11-30T21:51:10.369 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:11.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:09.526370+0000 mgr.smithi027.brrjcl (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:11.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:10.363624+0000 mon.smithi027 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.49:0/3725575131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:11.774 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:11.774 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:13.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.013+0000 7f4fce6b7700 1 -- 172.21.15.49:0/65064136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc80f5410 msgr2=0x7f4fc80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:13.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.013+0000 7f4fce6b7700 1 --2- 172.21.15.49:0/65064136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc80f5410 0x7f4fc80f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f4fb8009fb0 tx=0x7f4fb8009b30).stop 2023-11-30T21:51:13.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.013+0000 7f4fce6b7700 1 -- 172.21.15.49:0/65064136 shutdown_connections 2023-11-30T21:51:13.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.013+0000 7f4fce6b7700 1 --2- 172.21.15.49:0/65064136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc80f5410 0x7f4fc80f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:13.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.014+0000 7f4fce6b7700 1 -- 172.21.15.49:0/65064136 >> 172.21.15.49:0/65064136 conn(0x7f4fc80f0b70 msgr2=0x7f4fc80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:13.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.014+0000 7f4fce6b7700 1 -- 172.21.15.49:0/65064136 shutdown_connections 2023-11-30T21:51:13.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.014+0000 7f4fce6b7700 1 -- 172.21.15.49:0/65064136 wait complete. 2023-11-30T21:51:13.015 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.014+0000 7f4fce6b7700 1 Processor -- start 2023-11-30T21:51:13.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.014+0000 7f4fce6b7700 1 -- start start 2023-11-30T21:51:13.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.014+0000 7f4fce6b7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc80f5410 0x7f4fc8108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:13.016 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.014+0000 7f4fce6b7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc8108e00 con 0x7f4fc80f5410 2023-11-30T21:51:13.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.015+0000 7f4fc7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc80f5410 0x7f4fc8108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:13.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.015+0000 7f4fc7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc80f5410 0x7f4fc8108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:58484/0 (socket says 172.21.15.49:58484) 2023-11-30T21:51:13.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.015+0000 7f4fc7fff700 1 -- 172.21.15.49:0/3629282036 learned_addr learned my addr 172.21.15.49:0/3629282036 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:13.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.015+0000 7f4fc7fff700 1 -- 172.21.15.49:0/3629282036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb8005040 con 0x7f4fc80f5410 2023-11-30T21:51:13.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.015+0000 7f4fc7fff700 1 --2- 172.21.15.49:0/3629282036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc80f5410 0x7f4fc8108910 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f4fb8013510 tx=0x7f4fb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:13.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.015+0000 7f4fc4ff9700 1 -- 172.21.15.49:0/3629282036 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fb8009320 con 0x7f4fc80f5410 2023-11-30T21:51:13.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.016+0000 7f4fce6b7700 1 -- 172.21.15.49:0/3629282036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc8109000 con 0x7f4fc80f5410 2023-11-30T21:51:13.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.016+0000 7f4fc4ff9700 1 -- 172.21.15.49:0/3629282036 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4fb8013f00 con 0x7f4fc80f5410 2023-11-30T21:51:13.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.016+0000 7f4fc4ff9700 1 -- 172.21.15.49:0/3629282036 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fb8014d90 con 0x7f4fc80f5410 2023-11-30T21:51:13.018 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.016+0000 7f4fce6b7700 1 -- 172.21.15.49:0/3629282036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc8109490 con 0x7f4fc80f5410 2023-11-30T21:51:13.019 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.016+0000 7f4fce6b7700 1 -- 172.21.15.49:0/3629282036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc8045ad0 con 0x7f4fc80f5410 2023-11-30T21:51:13.019 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.016+0000 7f4fc4ff9700 1 -- 172.21.15.49:0/3629282036 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4fb80095f0 con 0x7f4fc80f5410 2023-11-30T21:51:13.019 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.017+0000 7f4fc4ff9700 1 --2- 172.21.15.49:0/3629282036 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4fb002ff60 0x7f4fb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:13.019 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.017+0000 7f4fc4ff9700 1 -- 172.21.15.49:0/3629282036 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4fb8040fe0 con 0x7f4fc80f5410 2023-11-30T21:51:13.019 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.017+0000 7f4fc77fe700 1 --2- 172.21.15.49:0/3629282036 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4fb002ff60 0x7f4fb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:13.019 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.017+0000 7f4fc77fe700 1 --2- 172.21.15.49:0/3629282036 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4fb002ff60 0x7f4fb0032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4fbc0096a0 tx=0x7f4fbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:13.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.021+0000 7f4fc4ff9700 1 -- 172.21.15.49:0/3629282036 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4fb801d7f0 con 0x7f4fc80f5410 2023-11-30T21:51:13.240 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.240+0000 7f4fce6b7700 1 -- 172.21.15.49:0/3629282036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4fc80fae60 con 0x7f4fc80f5410 2023-11-30T21:51:13.241 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.241+0000 7f4fc4ff9700 1 -- 172.21.15.49:0/3629282036 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4fb8010230 con 0x7f4fc80f5410 2023-11-30T21:51:13.241 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:13.242 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:13.243 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.242+0000 7f4fce6b7700 1 -- 172.21.15.49:0/3629282036 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4fb002ff60 msgr2=0x7f4fb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:13.243 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.242+0000 7f4fce6b7700 1 --2- 172.21.15.49:0/3629282036 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4fb002ff60 0x7f4fb0032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4fbc0096a0 tx=0x7f4fbc006b40).stop 2023-11-30T21:51:13.243 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.242+0000 7f4fce6b7700 1 -- 172.21.15.49:0/3629282036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc80f5410 msgr2=0x7f4fc8108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:13.243 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.242+0000 7f4fce6b7700 1 --2- 172.21.15.49:0/3629282036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc80f5410 0x7f4fc8108910 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f4fb8013510 tx=0x7f4fb800b040).stop 2023-11-30T21:51:13.243 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.242+0000 7f4fce6b7700 1 -- 172.21.15.49:0/3629282036 shutdown_connections 2023-11-30T21:51:13.243 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.242+0000 7f4fce6b7700 1 --2- 172.21.15.49:0/3629282036 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f4fb002ff60 0x7f4fb0032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:13.243 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.242+0000 7f4fce6b7700 1 --2- 172.21.15.49:0/3629282036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fc80f5410 0x7f4fc8108910 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:13.244 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.242+0000 7f4fce6b7700 1 -- 172.21.15.49:0/3629282036 >> 172.21.15.49:0/3629282036 conn(0x7f4fc80f0b70 msgr2=0x7f4fc805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:13.244 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.242+0000 7f4fce6b7700 1 -- 172.21.15.49:0/3629282036 shutdown_connections 2023-11-30T21:51:13.244 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:13.242+0000 7f4fce6b7700 1 -- 172.21.15.49:0/3629282036 wait complete. 2023-11-30T21:51:13.244 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:13.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:13 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:11.526730+0000 mgr.smithi027.brrjcl (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:14.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:14 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:13.241074+0000 mon.smithi027 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.49:0/3629282036' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:14.664 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:14.664 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:15.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:15 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:13.527079+0000 mgr.smithi027.brrjcl (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:15.916 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.915+0000 7f876359e700 1 -- 172.21.15.49:0/1976650876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87640f51a0 msgr2=0x7f87640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:15.916 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.915+0000 7f876359e700 1 --2- 172.21.15.49:0/1976650876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87640f51a0 0x7f87640f55c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f8754009fb0 tx=0x7f8754009b30).stop 2023-11-30T21:51:15.916 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.915+0000 7f876359e700 1 -- 172.21.15.49:0/1976650876 shutdown_connections 2023-11-30T21:51:15.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.915+0000 7f876359e700 1 --2- 172.21.15.49:0/1976650876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87640f51a0 0x7f87640f55c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:15.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.915+0000 7f876359e700 1 -- 172.21.15.49:0/1976650876 >> 172.21.15.49:0/1976650876 conn(0x7f87640f09a0 msgr2=0x7f87640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:15.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.915+0000 7f876359e700 1 -- 172.21.15.49:0/1976650876 shutdown_connections 2023-11-30T21:51:15.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.915+0000 7f876359e700 1 -- 172.21.15.49:0/1976650876 wait complete. 2023-11-30T21:51:15.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.916+0000 7f876359e700 1 Processor -- start 2023-11-30T21:51:15.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.916+0000 7f876359e700 1 -- start start 2023-11-30T21:51:15.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.916+0000 7f876359e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87640f51a0 0x7f87641091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:15.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.916+0000 7f876359e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8764060240 con 0x7f87640f51a0 2023-11-30T21:51:15.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.916+0000 7f876259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87640f51a0 0x7f87641091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:15.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.916+0000 7f876259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87640f51a0 0x7f87641091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:58512/0 (socket says 172.21.15.49:58512) 2023-11-30T21:51:15.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.916+0000 7f876259c700 1 -- 172.21.15.49:0/1584493800 learned_addr learned my addr 172.21.15.49:0/1584493800 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:15.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.916+0000 7f876259c700 1 -- 172.21.15.49:0/1584493800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8754005040 con 0x7f87640f51a0 2023-11-30T21:51:15.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.917+0000 7f876259c700 1 --2- 172.21.15.49:0/1584493800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87640f51a0 0x7f87641091b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f8754013510 tx=0x7f875400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:15.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.917+0000 7f8752ffd700 1 -- 172.21.15.49:0/1584493800 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8754009320 con 0x7f87640f51a0 2023-11-30T21:51:15.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.917+0000 7f876359e700 1 -- 172.21.15.49:0/1584493800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8764060440 con 0x7f87640f51a0 2023-11-30T21:51:15.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.917+0000 7f8752ffd700 1 -- 172.21.15.49:0/1584493800 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8754013f00 con 0x7f87640f51a0 2023-11-30T21:51:15.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.917+0000 7f8752ffd700 1 -- 172.21.15.49:0/1584493800 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8754014dc0 con 0x7f87640f51a0 2023-11-30T21:51:15.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.917+0000 7f876359e700 1 -- 172.21.15.49:0/1584493800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8764109860 con 0x7f87640f51a0 2023-11-30T21:51:15.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.918+0000 7f8752ffd700 1 -- 172.21.15.49:0/1584493800 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f875401d3d0 con 0x7f87640f51a0 2023-11-30T21:51:15.919 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.918+0000 7f8752ffd700 1 --2- 172.21.15.49:0/1584493800 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f874c02ff60 0x7f874c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:15.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.918+0000 7f876359e700 1 -- 172.21.15.49:0/1584493800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8764045ad0 con 0x7f87640f51a0 2023-11-30T21:51:15.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.918+0000 7f8752ffd700 1 -- 172.21.15.49:0/1584493800 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87540413e0 con 0x7f87640f51a0 2023-11-30T21:51:15.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.918+0000 7f8761d9b700 1 --2- 172.21.15.49:0/1584493800 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f874c02ff60 0x7f874c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:15.920 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.918+0000 7f8761d9b700 1 --2- 172.21.15.49:0/1584493800 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f874c02ff60 0x7f874c032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f87580096a0 tx=0x7f8758006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:15.924 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:15.923+0000 7f8752ffd700 1 -- 172.21.15.49:0/1584493800 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8754026490 con 0x7f87640f51a0 2023-11-30T21:51:16.146 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:16.143+0000 7f876359e700 1 -- 172.21.15.49:0/1584493800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f876410a070 con 0x7f87640f51a0 2023-11-30T21:51:16.147 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:16.144+0000 7f8752ffd700 1 -- 172.21.15.49:0/1584493800 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f875401cc70 con 0x7f87640f51a0 2023-11-30T21:51:16.147 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:16.147 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:16.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:16.145+0000 7f876359e700 1 -- 172.21.15.49:0/1584493800 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f874c02ff60 msgr2=0x7f874c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:16.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:16.145+0000 7f876359e700 1 --2- 172.21.15.49:0/1584493800 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f874c02ff60 0x7f874c032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f87580096a0 tx=0x7f8758006b40).stop 2023-11-30T21:51:16.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:16.145+0000 7f876359e700 1 -- 172.21.15.49:0/1584493800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87640f51a0 msgr2=0x7f87641091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:16.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:16.145+0000 7f876359e700 1 --2- 172.21.15.49:0/1584493800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87640f51a0 0x7f87641091b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f8754013510 tx=0x7f875400b040).stop 2023-11-30T21:51:16.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:16.145+0000 7f876359e700 1 -- 172.21.15.49:0/1584493800 shutdown_connections 2023-11-30T21:51:16.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:16.145+0000 7f876359e700 1 --2- 172.21.15.49:0/1584493800 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f874c02ff60 0x7f874c032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:16.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:16.145+0000 7f876359e700 1 --2- 172.21.15.49:0/1584493800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87640f51a0 0x7f87641091b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:16.148 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:16.145+0000 7f876359e700 1 -- 172.21.15.49:0/1584493800 >> 172.21.15.49:0/1584493800 conn(0x7f87640f09a0 msgr2=0x7f87640f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:16.149 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:16.145+0000 7f876359e700 1 -- 172.21.15.49:0/1584493800 shutdown_connections 2023-11-30T21:51:16.149 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:16.145+0000 7f876359e700 1 -- 172.21.15.49:0/1584493800 wait complete. 2023-11-30T21:51:16.149 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:16.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:16 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:16.144225+0000 mon.smithi027 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.49:0/1584493800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:17.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:15.527412+0000 mgr.smithi027.brrjcl (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:17.568 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:17.569 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:18.830 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.828+0000 7fa81289d700 1 -- 172.21.15.49:0/1280407954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa80c0f51a0 msgr2=0x7fa80c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:18.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.828+0000 7fa81289d700 1 --2- 172.21.15.49:0/1280407954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa80c0f51a0 0x7fa80c0f55c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fa808009fb0 tx=0x7fa808009b30).stop 2023-11-30T21:51:18.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.828+0000 7fa81289d700 1 -- 172.21.15.49:0/1280407954 shutdown_connections 2023-11-30T21:51:18.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.828+0000 7fa81289d700 1 --2- 172.21.15.49:0/1280407954 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa80c0f51a0 0x7fa80c0f55c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:18.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.828+0000 7fa81289d700 1 -- 172.21.15.49:0/1280407954 >> 172.21.15.49:0/1280407954 conn(0x7fa80c0f09a0 msgr2=0x7fa80c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:18.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.828+0000 7fa81289d700 1 -- 172.21.15.49:0/1280407954 shutdown_connections 2023-11-30T21:51:18.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.828+0000 7fa81289d700 1 -- 172.21.15.49:0/1280407954 wait complete. 2023-11-30T21:51:18.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.829+0000 7fa81289d700 1 Processor -- start 2023-11-30T21:51:18.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.829+0000 7fa81289d700 1 -- start start 2023-11-30T21:51:18.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.829+0000 7fa81289d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa80c0f51a0 0x7fa80c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:18.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.829+0000 7fa81289d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c060240 con 0x7fa80c0f51a0 2023-11-30T21:51:18.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.829+0000 7fa81189b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa80c0f51a0 0x7fa80c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:18.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.829+0000 7fa81189b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa80c0f51a0 0x7fa80c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59080/0 (socket says 172.21.15.49:59080) 2023-11-30T21:51:18.831 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.829+0000 7fa81189b700 1 -- 172.21.15.49:0/669144613 learned_addr learned my addr 172.21.15.49:0/669144613 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:18.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.829+0000 7fa81189b700 1 -- 172.21.15.49:0/669144613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa808005040 con 0x7fa80c0f51a0 2023-11-30T21:51:18.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.830+0000 7fa81189b700 1 --2- 172.21.15.49:0/669144613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa80c0f51a0 0x7fa80c1091b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa808013510 tx=0x7fa80800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:18.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.830+0000 7fa8027fc700 1 -- 172.21.15.49:0/669144613 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa808009320 con 0x7fa80c0f51a0 2023-11-30T21:51:18.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.830+0000 7fa8027fc700 1 -- 172.21.15.49:0/669144613 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa808013f00 con 0x7fa80c0f51a0 2023-11-30T21:51:18.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.830+0000 7fa81289d700 1 -- 172.21.15.49:0/669144613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80c060440 con 0x7fa80c0f51a0 2023-11-30T21:51:18.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.830+0000 7fa8027fc700 1 -- 172.21.15.49:0/669144613 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa808014dc0 con 0x7fa80c0f51a0 2023-11-30T21:51:18.832 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.830+0000 7fa81289d700 1 -- 172.21.15.49:0/669144613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa80c109860 con 0x7fa80c0f51a0 2023-11-30T21:51:18.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.831+0000 7fa8027fc700 1 -- 172.21.15.49:0/669144613 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8080095f0 con 0x7fa80c0f51a0 2023-11-30T21:51:18.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.831+0000 7fa8027fc700 1 --2- 172.21.15.49:0/669144613 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa7f802ff60 0x7fa7f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:18.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.831+0000 7fa81289d700 1 -- 172.21.15.49:0/669144613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80c045ad0 con 0x7fa80c0f51a0 2023-11-30T21:51:18.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.831+0000 7fa8027fc700 1 -- 172.21.15.49:0/669144613 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa808041040 con 0x7fa80c0f51a0 2023-11-30T21:51:18.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.831+0000 7fa81109a700 1 --2- 172.21.15.49:0/669144613 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa7f802ff60 0x7fa7f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:18.833 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.831+0000 7fa81109a700 1 --2- 172.21.15.49:0/669144613 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa7f802ff60 0x7fa7f8032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fa7fc0096a0 tx=0x7fa7fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:18.836 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:18.836+0000 7fa8027fc700 1 -- 172.21.15.49:0/669144613 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa80801c790 con 0x7fa80c0f51a0 2023-11-30T21:51:19.056 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:19.054+0000 7fa81289d700 1 -- 172.21.15.49:0/669144613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa80c10a080 con 0x7fa80c0f51a0 2023-11-30T21:51:19.056 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:19.055+0000 7fa8027fc700 1 -- 172.21.15.49:0/669144613 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa808010230 con 0x7fa80c0f51a0 2023-11-30T21:51:19.057 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:19.057 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:19.058 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:19.057+0000 7fa81289d700 1 -- 172.21.15.49:0/669144613 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa7f802ff60 msgr2=0x7fa7f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:19.058 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:19.057+0000 7fa81289d700 1 --2- 172.21.15.49:0/669144613 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa7f802ff60 0x7fa7f8032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fa7fc0096a0 tx=0x7fa7fc006b40).stop 2023-11-30T21:51:19.058 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:19.057+0000 7fa81289d700 1 -- 172.21.15.49:0/669144613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa80c0f51a0 msgr2=0x7fa80c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:19.059 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:19.057+0000 7fa81289d700 1 --2- 172.21.15.49:0/669144613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa80c0f51a0 0x7fa80c1091b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa808013510 tx=0x7fa80800b040).stop 2023-11-30T21:51:19.059 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:19.057+0000 7fa81289d700 1 -- 172.21.15.49:0/669144613 shutdown_connections 2023-11-30T21:51:19.059 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:19.057+0000 7fa81289d700 1 --2- 172.21.15.49:0/669144613 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa7f802ff60 0x7fa7f8032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:19.059 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:19.057+0000 7fa81289d700 1 --2- 172.21.15.49:0/669144613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa80c0f51a0 0x7fa80c1091b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:19.059 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:19.057+0000 7fa81289d700 1 -- 172.21.15.49:0/669144613 >> 172.21.15.49:0/669144613 conn(0x7fa80c0f09a0 msgr2=0x7fa80c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:19.059 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:19.057+0000 7fa81289d700 1 -- 172.21.15.49:0/669144613 shutdown_connections 2023-11-30T21:51:19.060 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:19.057+0000 7fa81289d700 1 -- 172.21.15.49:0/669144613 wait complete. 2023-11-30T21:51:19.060 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:19.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:17.527736+0000 mgr.smithi027.brrjcl (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:19.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:19.055635+0000 mon.smithi027 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.49:0/669144613' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:20.515 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:20.515 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:21.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:21 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:19.528104+0000 mgr.smithi027.brrjcl (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.818+0000 7fa2c0e75700 1 -- 172.21.15.49:0/1403726895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc0f51a0 msgr2=0x7fa2bc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.818+0000 7fa2c0e75700 1 --2- 172.21.15.49:0/1403726895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc0f51a0 0x7fa2bc0f55c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fa2ac009fb0 tx=0x7fa2ac009b30).stop 2023-11-30T21:51:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.818+0000 7fa2c0e75700 1 -- 172.21.15.49:0/1403726895 shutdown_connections 2023-11-30T21:51:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.818+0000 7fa2c0e75700 1 --2- 172.21.15.49:0/1403726895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc0f51a0 0x7fa2bc0f55c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.818+0000 7fa2c0e75700 1 -- 172.21.15.49:0/1403726895 >> 172.21.15.49:0/1403726895 conn(0x7fa2bc0f09a0 msgr2=0x7fa2bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:21.820 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.818+0000 7fa2c0e75700 1 -- 172.21.15.49:0/1403726895 shutdown_connections 2023-11-30T21:51:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.818+0000 7fa2c0e75700 1 -- 172.21.15.49:0/1403726895 wait complete. 2023-11-30T21:51:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.818+0000 7fa2c0e75700 1 Processor -- start 2023-11-30T21:51:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.819+0000 7fa2c0e75700 1 -- start start 2023-11-30T21:51:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.819+0000 7fa2c0e75700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc0f51a0 0x7fa2bc1064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.819+0000 7fa2c0e75700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc1069e0 con 0x7fa2bc0f51a0 2023-11-30T21:51:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.819+0000 7fa2bb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc0f51a0 0x7fa2bc1064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.819+0000 7fa2bb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc0f51a0 0x7fa2bc1064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59084/0 (socket says 172.21.15.49:59084) 2023-11-30T21:51:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.819+0000 7fa2bb7fe700 1 -- 172.21.15.49:0/936794107 learned_addr learned my addr 172.21.15.49:0/936794107 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.819+0000 7fa2bb7fe700 1 -- 172.21.15.49:0/936794107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ac005040 con 0x7fa2bc0f51a0 2023-11-30T21:51:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.819+0000 7fa2bb7fe700 1 --2- 172.21.15.49:0/936794107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc0f51a0 0x7fa2bc1064f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fa2ac009830 tx=0x7fa2ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.820+0000 7fa2a3fff700 1 -- 172.21.15.49:0/936794107 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2ac009320 con 0x7fa2bc0f51a0 2023-11-30T21:51:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.820+0000 7fa2a3fff700 1 -- 172.21.15.49:0/936794107 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2ac0136c0 con 0x7fa2bc0f51a0 2023-11-30T21:51:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.820+0000 7fa2c0e75700 1 -- 172.21.15.49:0/936794107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc106be0 con 0x7fa2bc0f51a0 2023-11-30T21:51:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.820+0000 7fa2a3fff700 1 -- 172.21.15.49:0/936794107 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2ac01ee90 con 0x7fa2bc0f51a0 2023-11-30T21:51:21.823 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.820+0000 7fa2c0e75700 1 -- 172.21.15.49:0/936794107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc107070 con 0x7fa2bc0f51a0 2023-11-30T21:51:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.821+0000 7fa2a3fff700 1 -- 172.21.15.49:0/936794107 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2ac01e420 con 0x7fa2bc0f51a0 2023-11-30T21:51:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.821+0000 7fa2c0e75700 1 -- 172.21.15.49:0/936794107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2bc045ad0 con 0x7fa2bc0f51a0 2023-11-30T21:51:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.821+0000 7fa2a3fff700 1 --2- 172.21.15.49:0/936794107 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa2a402ff10 0x7fa2a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.821+0000 7fa2a3fff700 1 -- 172.21.15.49:0/936794107 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2ac040f40 con 0x7fa2bc0f51a0 2023-11-30T21:51:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.821+0000 7fa2baffd700 1 --2- 172.21.15.49:0/936794107 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa2a402ff10 0x7fa2a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:21.824 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.822+0000 7fa2baffd700 1 --2- 172.21.15.49:0/936794107 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa2a402ff10 0x7fa2a40323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa2b00096a0 tx=0x7fa2b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:21.827 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:21.826+0000 7fa2a3fff700 1 -- 172.21.15.49:0/936794107 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2ac06d2b0 con 0x7fa2bc0f51a0 2023-11-30T21:51:22.047 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:22.045+0000 7fa2c0e75700 1 -- 172.21.15.49:0/936794107 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2bc108690 con 0x7fa2bc0f51a0 2023-11-30T21:51:22.047 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:22.046+0000 7fa2a3fff700 1 -- 172.21.15.49:0/936794107 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa2ac0095f0 con 0x7fa2bc0f51a0 2023-11-30T21:51:22.048 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:22.048 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:22.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:22.047+0000 7fa2c0e75700 1 -- 172.21.15.49:0/936794107 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa2a402ff10 msgr2=0x7fa2a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:22.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:22.047+0000 7fa2c0e75700 1 --2- 172.21.15.49:0/936794107 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa2a402ff10 0x7fa2a40323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa2b00096a0 tx=0x7fa2b0006b40).stop 2023-11-30T21:51:22.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:22.047+0000 7fa2c0e75700 1 -- 172.21.15.49:0/936794107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc0f51a0 msgr2=0x7fa2bc1064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:22.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:22.047+0000 7fa2c0e75700 1 --2- 172.21.15.49:0/936794107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc0f51a0 0x7fa2bc1064f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fa2ac009830 tx=0x7fa2ac00b040).stop 2023-11-30T21:51:22.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:22.047+0000 7fa2c0e75700 1 -- 172.21.15.49:0/936794107 shutdown_connections 2023-11-30T21:51:22.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:22.047+0000 7fa2c0e75700 1 --2- 172.21.15.49:0/936794107 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa2a402ff10 0x7fa2a40323d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:22.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:22.047+0000 7fa2c0e75700 1 --2- 172.21.15.49:0/936794107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2bc0f51a0 0x7fa2bc1064f0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:22.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:22.047+0000 7fa2c0e75700 1 -- 172.21.15.49:0/936794107 >> 172.21.15.49:0/936794107 conn(0x7fa2bc0f09a0 msgr2=0x7fa2bc05fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:22.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:22.047+0000 7fa2c0e75700 1 -- 172.21.15.49:0/936794107 shutdown_connections 2023-11-30T21:51:22.049 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:22.047+0000 7fa2c0e75700 1 -- 172.21.15.49:0/936794107 wait complete. 2023-11-30T21:51:22.050 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:22.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:22 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:22.047389+0000 mon.smithi027 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.49:0/936794107' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:23.488 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:23.488 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:23.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:23 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:21.528363+0000 mgr.smithi027.brrjcl (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:24.758 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.756+0000 7eff3dbad700 1 -- 172.21.15.49:0/168818823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380f6510 msgr2=0x7eff380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.756+0000 7eff3dbad700 1 --2- 172.21.15.49:0/168818823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380f6510 0x7eff380f6930 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7eff2c009fb0 tx=0x7eff2c009b30).stop 2023-11-30T21:51:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.756+0000 7eff3dbad700 1 -- 172.21.15.49:0/168818823 shutdown_connections 2023-11-30T21:51:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.756+0000 7eff3dbad700 1 --2- 172.21.15.49:0/168818823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380f6510 0x7eff380f6930 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.756+0000 7eff3dbad700 1 -- 172.21.15.49:0/168818823 >> 172.21.15.49:0/168818823 conn(0x7eff380f1c70 msgr2=0x7eff380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.756+0000 7eff3dbad700 1 -- 172.21.15.49:0/168818823 shutdown_connections 2023-11-30T21:51:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.756+0000 7eff3dbad700 1 -- 172.21.15.49:0/168818823 wait complete. 2023-11-30T21:51:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.756+0000 7eff3dbad700 1 Processor -- start 2023-11-30T21:51:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.757+0000 7eff3dbad700 1 -- start start 2023-11-30T21:51:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.757+0000 7eff3dbad700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380f6510 0x7eff38106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.757+0000 7eff3dbad700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff38106c20 con 0x7eff380f6510 2023-11-30T21:51:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.757+0000 7eff377fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380f6510 0x7eff38106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:24.759 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.757+0000 7eff377fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380f6510 0x7eff38106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59110/0 (socket says 172.21.15.49:59110) 2023-11-30T21:51:24.760 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.757+0000 7eff377fe700 1 -- 172.21.15.49:0/3461463080 learned_addr learned my addr 172.21.15.49:0/3461463080 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:24.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.757+0000 7eff377fe700 1 -- 172.21.15.49:0/3461463080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2c005040 con 0x7eff380f6510 2023-11-30T21:51:24.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.758+0000 7eff377fe700 1 --2- 172.21.15.49:0/3461463080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380f6510 0x7eff38106730 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7eff2c013510 tx=0x7eff2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:24.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.758+0000 7eff1ffff700 1 -- 172.21.15.49:0/3461463080 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2c009320 con 0x7eff380f6510 2023-11-30T21:51:24.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.758+0000 7eff3dbad700 1 -- 172.21.15.49:0/3461463080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff38106e20 con 0x7eff380f6510 2023-11-30T21:51:24.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.758+0000 7eff1ffff700 1 -- 172.21.15.49:0/3461463080 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff2c013f00 con 0x7eff380f6510 2023-11-30T21:51:24.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.758+0000 7eff1ffff700 1 -- 172.21.15.49:0/3461463080 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2c014d90 con 0x7eff380f6510 2023-11-30T21:51:24.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.758+0000 7eff3dbad700 1 -- 172.21.15.49:0/3461463080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff381072b0 con 0x7eff380f6510 2023-11-30T21:51:24.761 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.758+0000 7eff1ffff700 1 -- 172.21.15.49:0/3461463080 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7eff2c0095f0 con 0x7eff380f6510 2023-11-30T21:51:24.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.759+0000 7eff1ffff700 1 --2- 172.21.15.49:0/3461463080 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7eff2002ff60 0x7eff20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:24.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.759+0000 7eff3dbad700 1 -- 172.21.15.49:0/3461463080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff38045ad0 con 0x7eff380f6510 2023-11-30T21:51:24.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.759+0000 7eff1ffff700 1 -- 172.21.15.49:0/3461463080 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7eff2c0410c0 con 0x7eff380f6510 2023-11-30T21:51:24.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.759+0000 7eff36ffd700 1 --2- 172.21.15.49:0/3461463080 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7eff2002ff60 0x7eff20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:24.762 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.759+0000 7eff36ffd700 1 --2- 172.21.15.49:0/3461463080 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7eff2002ff60 0x7eff20032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7eff280096a0 tx=0x7eff28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:24.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.764+0000 7eff1ffff700 1 -- 172.21.15.49:0/3461463080 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff2c06d2b0 con 0x7eff380f6510 2023-11-30T21:51:24.984 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.982+0000 7eff3dbad700 1 -- 172.21.15.49:0/3461463080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff380fc050 con 0x7eff380f6510 2023-11-30T21:51:24.985 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.983+0000 7eff1ffff700 1 -- 172.21.15.49:0/3461463080 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7eff2c014420 con 0x7eff380f6510 2023-11-30T21:51:24.985 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:24.985 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:24.986 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.985+0000 7eff3dbad700 1 -- 172.21.15.49:0/3461463080 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7eff2002ff60 msgr2=0x7eff20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:24.986 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.985+0000 7eff3dbad700 1 --2- 172.21.15.49:0/3461463080 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7eff2002ff60 0x7eff20032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7eff280096a0 tx=0x7eff28006b40).stop 2023-11-30T21:51:24.986 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.985+0000 7eff3dbad700 1 -- 172.21.15.49:0/3461463080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380f6510 msgr2=0x7eff38106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:24.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.985+0000 7eff3dbad700 1 --2- 172.21.15.49:0/3461463080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380f6510 0x7eff38106730 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7eff2c013510 tx=0x7eff2c00b040).stop 2023-11-30T21:51:24.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.985+0000 7eff3dbad700 1 -- 172.21.15.49:0/3461463080 shutdown_connections 2023-11-30T21:51:24.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.985+0000 7eff3dbad700 1 --2- 172.21.15.49:0/3461463080 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7eff2002ff60 0x7eff20032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:24.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.985+0000 7eff3dbad700 1 --2- 172.21.15.49:0/3461463080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380f6510 0x7eff38106730 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:24.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.985+0000 7eff3dbad700 1 -- 172.21.15.49:0/3461463080 >> 172.21.15.49:0/3461463080 conn(0x7eff380f1c70 msgr2=0x7eff380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:24.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.985+0000 7eff3dbad700 1 -- 172.21.15.49:0/3461463080 shutdown_connections 2023-11-30T21:51:24.987 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:24.985+0000 7eff3dbad700 1 -- 172.21.15.49:0/3461463080 wait complete. 2023-11-30T21:51:24.987 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:25.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:23.528653+0000 mgr.smithi027.brrjcl (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:25.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:24.984662+0000 mon.smithi027 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.49:0/3461463080' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:26.374 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:26.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:27.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:27 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:25.529054+0000 mgr.smithi027.brrjcl (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:27.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.625+0000 7fdb52f96700 1 -- 172.21.15.49:0/4200588377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb4c060220 msgr2=0x7fdb4c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:27.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.625+0000 7fdb52f96700 1 --2- 172.21.15.49:0/4200588377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb4c060220 0x7fdb4c0f9150 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fdb40009fb0 tx=0x7fdb40009b30).stop 2023-11-30T21:51:27.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.625+0000 7fdb52f96700 1 -- 172.21.15.49:0/4200588377 shutdown_connections 2023-11-30T21:51:27.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.625+0000 7fdb52f96700 1 --2- 172.21.15.49:0/4200588377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb4c060220 0x7fdb4c0f9150 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:27.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.625+0000 7fdb52f96700 1 -- 172.21.15.49:0/4200588377 >> 172.21.15.49:0/4200588377 conn(0x7fdb4c0f0b70 msgr2=0x7fdb4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:27.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.625+0000 7fdb52f96700 1 -- 172.21.15.49:0/4200588377 shutdown_connections 2023-11-30T21:51:27.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.625+0000 7fdb52f96700 1 -- 172.21.15.49:0/4200588377 wait complete. 2023-11-30T21:51:27.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.626+0000 7fdb52f96700 1 Processor -- start 2023-11-30T21:51:27.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.626+0000 7fdb52f96700 1 -- start start 2023-11-30T21:51:27.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.626+0000 7fdb52f96700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb4c060220 0x7fdb4c0f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:27.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.626+0000 7fdb52f96700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c0f5050 con 0x7fdb4c060220 2023-11-30T21:51:27.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.626+0000 7fdb50d32700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb4c060220 0x7fdb4c0f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:27.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.626+0000 7fdb50d32700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb4c060220 0x7fdb4c0f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:59122/0 (socket says 172.21.15.49:59122) 2023-11-30T21:51:27.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.626+0000 7fdb50d32700 1 -- 172.21.15.49:0/838779039 learned_addr learned my addr 172.21.15.49:0/838779039 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:27.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.627+0000 7fdb50d32700 1 -- 172.21.15.49:0/838779039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb40005040 con 0x7fdb4c060220 2023-11-30T21:51:27.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.627+0000 7fdb50d32700 1 --2- 172.21.15.49:0/838779039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb4c060220 0x7fdb4c0f6bd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fdb40013510 tx=0x7fdb4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:27.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.627+0000 7fdb497fa700 1 -- 172.21.15.49:0/838779039 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb40009320 con 0x7fdb4c060220 2023-11-30T21:51:27.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.627+0000 7fdb52f96700 1 -- 172.21.15.49:0/838779039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb4c0f51f0 con 0x7fdb4c060220 2023-11-30T21:51:27.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.627+0000 7fdb497fa700 1 -- 172.21.15.49:0/838779039 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb40013f00 con 0x7fdb4c060220 2023-11-30T21:51:27.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.627+0000 7fdb497fa700 1 -- 172.21.15.49:0/838779039 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb40014dc0 con 0x7fdb4c060220 2023-11-30T21:51:27.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.627+0000 7fdb52f96700 1 -- 172.21.15.49:0/838779039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4c0f5680 con 0x7fdb4c060220 2023-11-30T21:51:27.632 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.628+0000 7fdb497fa700 1 -- 172.21.15.49:0/838779039 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb400263a0 con 0x7fdb4c060220 2023-11-30T21:51:27.632 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.628+0000 7fdb497fa700 1 --2- 172.21.15.49:0/838779039 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdb3c02ffb0 0x7fdb3c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:27.632 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.628+0000 7fdb497fa700 1 -- 172.21.15.49:0/838779039 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb40040fd0 con 0x7fdb4c060220 2023-11-30T21:51:27.633 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.628+0000 7fdb52f96700 1 -- 172.21.15.49:0/838779039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb4c059100 con 0x7fdb4c060220 2023-11-30T21:51:27.633 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.628+0000 7fdb4bfff700 1 --2- 172.21.15.49:0/838779039 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdb3c02ffb0 0x7fdb3c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:27.633 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.628+0000 7fdb4bfff700 1 --2- 172.21.15.49:0/838779039 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdb3c02ffb0 0x7fdb3c032470 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fdb380096a0 tx=0x7fdb38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:27.634 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.633+0000 7fdb497fa700 1 -- 172.21.15.49:0/838779039 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb40012920 con 0x7fdb4c060220 2023-11-30T21:51:27.851 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.850+0000 7fdb52f96700 1 -- 172.21.15.49:0/838779039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb4c045ad0 con 0x7fdb4c060220 2023-11-30T21:51:27.852 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.850+0000 7fdb497fa700 1 -- 172.21.15.49:0/838779039 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdb4002b020 con 0x7fdb4c060220 2023-11-30T21:51:27.852 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:27.852 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:27.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.852+0000 7fdb52f96700 1 -- 172.21.15.49:0/838779039 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdb3c02ffb0 msgr2=0x7fdb3c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:27.853 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.852+0000 7fdb52f96700 1 --2- 172.21.15.49:0/838779039 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdb3c02ffb0 0x7fdb3c032470 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fdb380096a0 tx=0x7fdb38006b40).stop 2023-11-30T21:51:27.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.852+0000 7fdb52f96700 1 -- 172.21.15.49:0/838779039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb4c060220 msgr2=0x7fdb4c0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:27.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.852+0000 7fdb52f96700 1 --2- 172.21.15.49:0/838779039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb4c060220 0x7fdb4c0f6bd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fdb40013510 tx=0x7fdb4000b040).stop 2023-11-30T21:51:27.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.852+0000 7fdb52f96700 1 -- 172.21.15.49:0/838779039 shutdown_connections 2023-11-30T21:51:27.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.852+0000 7fdb52f96700 1 --2- 172.21.15.49:0/838779039 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdb3c02ffb0 0x7fdb3c032470 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:27.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.852+0000 7fdb52f96700 1 --2- 172.21.15.49:0/838779039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb4c060220 0x7fdb4c0f6bd0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:27.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.852+0000 7fdb52f96700 1 -- 172.21.15.49:0/838779039 >> 172.21.15.49:0/838779039 conn(0x7fdb4c0f0b70 msgr2=0x7fdb4c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:27.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.852+0000 7fdb52f96700 1 -- 172.21.15.49:0/838779039 shutdown_connections 2023-11-30T21:51:27.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:27.852+0000 7fdb52f96700 1 -- 172.21.15.49:0/838779039 wait complete. 2023-11-30T21:51:27.855 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:28.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:28 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:27.851824+0000 mon.smithi027 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.49:0/838779039' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:29.320 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:29.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:29.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:27.529377+0000 mgr.smithi027.brrjcl (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:30.573 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.561+0000 7f9dae553700 1 -- 172.21.15.49:0/1431941041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9da8060220 msgr2=0x7f9da80f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:30.574 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.561+0000 7f9dae553700 1 --2- 172.21.15.49:0/1431941041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9da8060220 0x7f9da80f9150 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f9d98009fb0 tx=0x7f9d98009b30).stop 2023-11-30T21:51:30.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.561+0000 7f9dae553700 1 -- 172.21.15.49:0/1431941041 shutdown_connections 2023-11-30T21:51:30.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.561+0000 7f9dae553700 1 --2- 172.21.15.49:0/1431941041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9da8060220 0x7f9da80f9150 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:30.576 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.561+0000 7f9dae553700 1 -- 172.21.15.49:0/1431941041 >> 172.21.15.49:0/1431941041 conn(0x7f9da80f0b70 msgr2=0x7f9da80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:30.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.561+0000 7f9dae553700 1 -- 172.21.15.49:0/1431941041 shutdown_connections 2023-11-30T21:51:30.577 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.561+0000 7f9dae553700 1 -- 172.21.15.49:0/1431941041 wait complete. 2023-11-30T21:51:30.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.561+0000 7f9dae553700 1 Processor -- start 2023-11-30T21:51:30.578 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.562+0000 7f9dae553700 1 -- start start 2023-11-30T21:51:30.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.562+0000 7f9dae553700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9da8060220 0x7f9da80f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:30.579 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.562+0000 7f9dae553700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da80f5050 con 0x7f9da8060220 2023-11-30T21:51:30.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.562+0000 7f9da7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9da8060220 0x7f9da80f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:30.580 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.562+0000 7f9da7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9da8060220 0x7f9da80f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:54702/0 (socket says 172.21.15.49:54702) 2023-11-30T21:51:30.581 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.562+0000 7f9da7fff700 1 -- 172.21.15.49:0/151659113 learned_addr learned my addr 172.21.15.49:0/151659113 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:30.581 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.562+0000 7f9da7fff700 1 -- 172.21.15.49:0/151659113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d98005040 con 0x7f9da8060220 2023-11-30T21:51:30.582 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.562+0000 7f9da7fff700 1 --2- 172.21.15.49:0/151659113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9da8060220 0x7f9da80f6bd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f9d98013510 tx=0x7f9d9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:30.583 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.563+0000 7f9da4ff9700 1 -- 172.21.15.49:0/151659113 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d98009320 con 0x7f9da8060220 2023-11-30T21:51:30.583 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.563+0000 7f9dae553700 1 -- 172.21.15.49:0/151659113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da80f51f0 con 0x7f9da8060220 2023-11-30T21:51:30.584 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.563+0000 7f9da4ff9700 1 -- 172.21.15.49:0/151659113 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d98013f00 con 0x7f9da8060220 2023-11-30T21:51:30.584 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.563+0000 7f9da4ff9700 1 -- 172.21.15.49:0/151659113 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d98014dc0 con 0x7f9da8060220 2023-11-30T21:51:30.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.563+0000 7f9dae553700 1 -- 172.21.15.49:0/151659113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da80f5680 con 0x7f9da8060220 2023-11-30T21:51:30.585 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.564+0000 7f9da4ff9700 1 -- 172.21.15.49:0/151659113 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d980095f0 con 0x7f9da8060220 2023-11-30T21:51:30.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.564+0000 7f9dae553700 1 -- 172.21.15.49:0/151659113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da8059100 con 0x7f9da8060220 2023-11-30T21:51:30.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.564+0000 7f9da4ff9700 1 --2- 172.21.15.49:0/151659113 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d9402ff60 0x7f9d94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:30.586 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.564+0000 7f9da4ff9700 1 -- 172.21.15.49:0/151659113 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d98041020 con 0x7f9da8060220 2023-11-30T21:51:30.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.564+0000 7f9da77fe700 1 --2- 172.21.15.49:0/151659113 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d9402ff60 0x7f9d94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:30.587 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.564+0000 7f9da77fe700 1 --2- 172.21.15.49:0/151659113 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d9402ff60 0x7f9d94032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f9d900096a0 tx=0x7f9d90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:30.588 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.569+0000 7f9da4ff9700 1 -- 172.21.15.49:0/151659113 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d98014420 con 0x7f9da8060220 2023-11-30T21:51:30.785 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.783+0000 7f9dae553700 1 -- 172.21.15.49:0/151659113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9da80fbf50 con 0x7f9da8060220 2023-11-30T21:51:30.785 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.784+0000 7f9da4ff9700 1 -- 172.21.15.49:0/151659113 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9d9801c920 con 0x7f9da8060220 2023-11-30T21:51:30.785 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:30.786 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:30.787 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.785+0000 7f9dae553700 1 -- 172.21.15.49:0/151659113 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d9402ff60 msgr2=0x7f9d94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:30.787 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.785+0000 7f9dae553700 1 --2- 172.21.15.49:0/151659113 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d9402ff60 0x7f9d94032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f9d900096a0 tx=0x7f9d90006b40).stop 2023-11-30T21:51:30.787 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.785+0000 7f9dae553700 1 -- 172.21.15.49:0/151659113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9da8060220 msgr2=0x7f9da80f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:30.787 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.785+0000 7f9dae553700 1 --2- 172.21.15.49:0/151659113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9da8060220 0x7f9da80f6bd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f9d98013510 tx=0x7f9d9800b040).stop 2023-11-30T21:51:30.787 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.785+0000 7f9dae553700 1 -- 172.21.15.49:0/151659113 shutdown_connections 2023-11-30T21:51:30.787 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.785+0000 7f9dae553700 1 --2- 172.21.15.49:0/151659113 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f9d9402ff60 0x7f9d94032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:30.787 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.785+0000 7f9dae553700 1 --2- 172.21.15.49:0/151659113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9da8060220 0x7f9da80f6bd0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:30.788 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.785+0000 7f9dae553700 1 -- 172.21.15.49:0/151659113 >> 172.21.15.49:0/151659113 conn(0x7f9da80f0b70 msgr2=0x7f9da80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:30.788 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.785+0000 7f9dae553700 1 -- 172.21.15.49:0/151659113 shutdown_connections 2023-11-30T21:51:30.788 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:30.785+0000 7f9dae553700 1 -- 172.21.15.49:0/151659113 wait complete. 2023-11-30T21:51:30.788 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:31.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:29.529663+0000 mgr.smithi027.brrjcl (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:31.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:30.785253+0000 mon.smithi027 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.49:0/151659113' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:32.187 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:32.188 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:33.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.464+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/3819592915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b40f6510 msgr2=0x7ff1b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:33.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.464+0000 7ff1ba6ad700 1 --2- 172.21.15.49:0/3819592915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b40f6510 0x7ff1b40f6930 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7ff1a4009fb0 tx=0x7ff1a4009b30).stop 2023-11-30T21:51:33.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.465+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/3819592915 shutdown_connections 2023-11-30T21:51:33.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.465+0000 7ff1ba6ad700 1 --2- 172.21.15.49:0/3819592915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b40f6510 0x7ff1b40f6930 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:33.468 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.465+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/3819592915 >> 172.21.15.49:0/3819592915 conn(0x7ff1b40f1c70 msgr2=0x7ff1b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:33.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.465+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/3819592915 shutdown_connections 2023-11-30T21:51:33.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.465+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/3819592915 wait complete. 2023-11-30T21:51:33.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.465+0000 7ff1ba6ad700 1 Processor -- start 2023-11-30T21:51:33.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.465+0000 7ff1ba6ad700 1 -- start start 2023-11-30T21:51:33.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.465+0000 7ff1ba6ad700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b40f6510 0x7ff1b406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:33.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.465+0000 7ff1ba6ad700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b4069e20 con 0x7ff1b40f6510 2023-11-30T21:51:33.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.466+0000 7ff1b3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b40f6510 0x7ff1b406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:33.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.466+0000 7ff1b3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b40f6510 0x7ff1b406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:54722/0 (socket says 172.21.15.49:54722) 2023-11-30T21:51:33.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.466+0000 7ff1b3fff700 1 -- 172.21.15.49:0/1891466272 learned_addr learned my addr 172.21.15.49:0/1891466272 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:33.469 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.466+0000 7ff1b3fff700 1 -- 172.21.15.49:0/1891466272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a4005040 con 0x7ff1b40f6510 2023-11-30T21:51:33.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.466+0000 7ff1b3fff700 1 --2- 172.21.15.49:0/1891466272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b40f6510 0x7ff1b406b9a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7ff1a4013510 tx=0x7ff1a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:33.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.466+0000 7ff1b0ff9700 1 -- 172.21.15.49:0/1891466272 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1a4009320 con 0x7ff1b40f6510 2023-11-30T21:51:33.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.466+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/1891466272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b4069fc0 con 0x7ff1b40f6510 2023-11-30T21:51:33.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.466+0000 7ff1b0ff9700 1 -- 172.21.15.49:0/1891466272 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1a4013f00 con 0x7ff1b40f6510 2023-11-30T21:51:33.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.466+0000 7ff1b0ff9700 1 -- 172.21.15.49:0/1891466272 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1a4014d90 con 0x7ff1b40f6510 2023-11-30T21:51:33.470 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.466+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/1891466272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b406a450 con 0x7ff1b40f6510 2023-11-30T21:51:33.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.467+0000 7ff1b0ff9700 1 -- 172.21.15.49:0/1891466272 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1a40095f0 con 0x7ff1b40f6510 2023-11-30T21:51:33.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.467+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/1891466272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b4059100 con 0x7ff1b40f6510 2023-11-30T21:51:33.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.467+0000 7ff1b0ff9700 1 --2- 172.21.15.49:0/1891466272 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff19c02ff60 0x7ff19c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:33.471 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.467+0000 7ff1b0ff9700 1 -- 172.21.15.49:0/1891466272 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff1a4040fc0 con 0x7ff1b40f6510 2023-11-30T21:51:33.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.468+0000 7ff1b37fe700 1 --2- 172.21.15.49:0/1891466272 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff19c02ff60 0x7ff19c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:33.472 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.468+0000 7ff1b37fe700 1 --2- 172.21.15.49:0/1891466272 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff19c02ff60 0x7ff19c032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff1a80096a0 tx=0x7ff1a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:33.474 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.472+0000 7ff1b0ff9700 1 -- 172.21.15.49:0/1891466272 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff1a4014420 con 0x7ff1b40f6510 2023-11-30T21:51:33.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:33 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:31.529987+0000 mgr.smithi027.brrjcl (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:33.694 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.692+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/1891466272 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1b4045ad0 con 0x7ff1b40f6510 2023-11-30T21:51:33.694 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.693+0000 7ff1b0ff9700 1 -- 172.21.15.49:0/1891466272 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff1a4010230 con 0x7ff1b40f6510 2023-11-30T21:51:33.695 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:33.695 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.694+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/1891466272 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff19c02ff60 msgr2=0x7ff19c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.694+0000 7ff1ba6ad700 1 --2- 172.21.15.49:0/1891466272 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff19c02ff60 0x7ff19c032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff1a80096a0 tx=0x7ff1a8006b40).stop 2023-11-30T21:51:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.694+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/1891466272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b40f6510 msgr2=0x7ff1b406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.694+0000 7ff1ba6ad700 1 --2- 172.21.15.49:0/1891466272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b40f6510 0x7ff1b406b9a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7ff1a4013510 tx=0x7ff1a400b040).stop 2023-11-30T21:51:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.694+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/1891466272 shutdown_connections 2023-11-30T21:51:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.694+0000 7ff1ba6ad700 1 --2- 172.21.15.49:0/1891466272 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff19c02ff60 0x7ff19c032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.694+0000 7ff1ba6ad700 1 --2- 172.21.15.49:0/1891466272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1b40f6510 0x7ff1b406b9a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.694+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/1891466272 >> 172.21.15.49:0/1891466272 conn(0x7ff1b40f1c70 msgr2=0x7ff1b40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.694+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/1891466272 shutdown_connections 2023-11-30T21:51:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:33.695+0000 7ff1ba6ad700 1 -- 172.21.15.49:0/1891466272 wait complete. 2023-11-30T21:51:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:34.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:34 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:33.694468+0000 mon.smithi027 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.49:0/1891466272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:35.072 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:35.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:35.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:35 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:33.530333+0000 mgr.smithi027.brrjcl (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:36.311 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.309+0000 7f1421c2f700 1 -- 172.21.15.49:0/1865752826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c0f5410 msgr2=0x7f141c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:36.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.309+0000 7f1421c2f700 1 --2- 172.21.15.49:0/1865752826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c0f5410 0x7f141c0f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f140c009fb0 tx=0x7f140c009b30).stop 2023-11-30T21:51:36.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.309+0000 7f1421c2f700 1 -- 172.21.15.49:0/1865752826 shutdown_connections 2023-11-30T21:51:36.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.309+0000 7f1421c2f700 1 --2- 172.21.15.49:0/1865752826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c0f5410 0x7f141c0f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:36.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.309+0000 7f1421c2f700 1 -- 172.21.15.49:0/1865752826 >> 172.21.15.49:0/1865752826 conn(0x7f141c0f0b70 msgr2=0x7f141c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:36.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.309+0000 7f1421c2f700 1 -- 172.21.15.49:0/1865752826 shutdown_connections 2023-11-30T21:51:36.312 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.309+0000 7f1421c2f700 1 -- 172.21.15.49:0/1865752826 wait complete. 2023-11-30T21:51:36.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.310+0000 7f1421c2f700 1 Processor -- start 2023-11-30T21:51:36.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.310+0000 7f1421c2f700 1 -- start start 2023-11-30T21:51:36.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.311+0000 7f1421c2f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c0f5410 0x7f141c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:36.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.311+0000 7f1421c2f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c060240 con 0x7f141c0f5410 2023-11-30T21:51:36.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.311+0000 7f141b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c0f5410 0x7f141c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:36.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.311+0000 7f141b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c0f5410 0x7f141c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:54738/0 (socket says 172.21.15.49:54738) 2023-11-30T21:51:36.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.311+0000 7f141b7fe700 1 -- 172.21.15.49:0/1963584369 learned_addr learned my addr 172.21.15.49:0/1963584369 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:36.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.311+0000 7f141b7fe700 1 -- 172.21.15.49:0/1963584369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140c005040 con 0x7f141c0f5410 2023-11-30T21:51:36.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.311+0000 7f141b7fe700 1 --2- 172.21.15.49:0/1963584369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c0f5410 0x7f141c109430 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f140c013510 tx=0x7f140c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:36.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.312+0000 7f1403fff700 1 -- 172.21.15.49:0/1963584369 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f140c009320 con 0x7f141c0f5410 2023-11-30T21:51:36.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.312+0000 7f1421c2f700 1 -- 172.21.15.49:0/1963584369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f141c060440 con 0x7f141c0f5410 2023-11-30T21:51:36.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.312+0000 7f1403fff700 1 -- 172.21.15.49:0/1963584369 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f140c013f00 con 0x7f141c0f5410 2023-11-30T21:51:36.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.312+0000 7f1403fff700 1 -- 172.21.15.49:0/1963584369 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f140c014d90 con 0x7f141c0f5410 2023-11-30T21:51:36.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.312+0000 7f1421c2f700 1 -- 172.21.15.49:0/1963584369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f141c109ae0 con 0x7f141c0f5410 2023-11-30T21:51:36.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.312+0000 7f1403fff700 1 -- 172.21.15.49:0/1963584369 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f140c014420 con 0x7f141c0f5410 2023-11-30T21:51:36.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.313+0000 7f1421c2f700 1 -- 172.21.15.49:0/1963584369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f141c045ad0 con 0x7f141c0f5410 2023-11-30T21:51:36.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.313+0000 7f1403fff700 1 --2- 172.21.15.49:0/1963584369 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f140402ff60 0x7f1404032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:36.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.313+0000 7f1403fff700 1 -- 172.21.15.49:0/1963584369 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f140c041000 con 0x7f141c0f5410 2023-11-30T21:51:36.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.313+0000 7f141affd700 1 --2- 172.21.15.49:0/1963584369 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f140402ff60 0x7f1404032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:36.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.313+0000 7f141affd700 1 --2- 172.21.15.49:0/1963584369 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f140402ff60 0x7f1404032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f14100096a0 tx=0x7f1410006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:36.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.317+0000 7f1403fff700 1 -- 172.21.15.49:0/1963584369 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f140c0095f0 con 0x7f141c0f5410 2023-11-30T21:51:36.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:51:34.551652+0000 mgr.smithi027.brrjcl (mgr.14162) 232 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi027 ... 2023-11-30T21:51:36.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Verifying port 9100 ... 2023-11-30T21:51:36.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Non-zero exit code 1 from systemctl start ceph-6473e080-8fc9-11ee-95a2-87774f69a715@node-exporter.smithi027 2023-11-30T21:51:36.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: systemctl: stderr Job for ceph-6473e080-8fc9-11ee-95a2-87774f69a715@node-exporter.smithi027.service failed because a timeout was exceeded. 2023-11-30T21:51:36.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: systemctl: stderr See "systemctl status ceph-6473e080-8fc9-11ee-95a2-87774f69a715@node-exporter.smithi027.service" and "journalctl -xe" for details. 2023-11-30T21:51:36.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Traceback (most recent call last): 2023-11-30T21:51:36.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-30T21:51:36.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: main() 2023-11-30T21:51:36.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-30T21:51:36.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: r = ctx.func(ctx) 2023-11-30T21:51:36.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-30T21:51:36.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: return func(ctx) 2023-11-30T21:51:36.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-30T21:51:36.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: ports=daemon_ports) 2023-11-30T21:51:36.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-30T21:51:36.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-30T21:51:36.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-30T21:51:36.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-30T21:51:36.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-30T21:51:36.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-30T21:51:36.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: RuntimeError: Failed command: systemctl start ceph-6473e080-8fc9-11ee-95a2-87774f69a715@node-exporter.smithi027 2023-11-30T21:51:36.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Traceback (most recent call last): 2023-11-30T21:51:36.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-30T21:51:36.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: yield (conn, connr) 2023-11-30T21:51:36.515 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-30T21:51:36.516 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: code, '\n'.join(err))) 2023-11-30T21:51:36.516 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi027 ... 2023-11-30T21:51:36.516 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Verifying port 9100 ... 2023-11-30T21:51:36.516 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Non-zero exit code 1 from systemctl start ceph-6473e080-8fc9-11ee-95a2-87774f69a715@node-exporter.smithi027 2023-11-30T21:51:36.516 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: systemctl: stderr Job for ceph-6473e080-8fc9-11ee-95a2-87774f69a715@node-exporter.smithi027.service failed because a timeout was exceeded. 2023-11-30T21:51:36.516 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: systemctl: stderr See "systemctl status ceph-6473e080-8fc9-11ee-95a2-87774f69a715@node-exporter.smithi027.service" and "journalctl -xe" for details. 2023-11-30T21:51:36.516 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: Traceback (most recent call last): 2023-11-30T21:51:36.517 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-30T21:51:36.517 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: main() 2023-11-30T21:51:36.517 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-30T21:51:36.517 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: r = ctx.func(ctx) 2023-11-30T21:51:36.517 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-30T21:51:36.517 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: return func(ctx) 2023-11-30T21:51:36.518 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-30T21:51:36.518 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: ports=daemon_ports) 2023-11-30T21:51:36.518 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-30T21:51:36.518 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-30T21:51:36.518 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-30T21:51:36.518 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-30T21:51:36.518 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: File "/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-30T21:51:36.519 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-30T21:51:36.519 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: RuntimeError: Failed command: systemctl start ceph-6473e080-8fc9-11ee-95a2-87774f69a715@node-exporter.smithi027 2023-11-30T21:51:36.519 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:36 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cephadm 2023-11-30T21:51:34.557379+0000 mgr.smithi027.brrjcl (mgr.14162) 233 : cephadm [INF] Deploying daemon prometheus.smithi027 on smithi027 2023-11-30T21:51:36.534 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.533+0000 7f1421c2f700 1 -- 172.21.15.49:0/1963584369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f141c0fae60 con 0x7f141c0f5410 2023-11-30T21:51:36.535 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.533+0000 7f1403fff700 1 -- 172.21.15.49:0/1963584369 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f140c01db70 con 0x7f141c0f5410 2023-11-30T21:51:36.535 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:36.535 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:36.536 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.535+0000 7f1421c2f700 1 -- 172.21.15.49:0/1963584369 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f140402ff60 msgr2=0x7f1404032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:36.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.535+0000 7f1421c2f700 1 --2- 172.21.15.49:0/1963584369 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f140402ff60 0x7f1404032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f14100096a0 tx=0x7f1410006b40).stop 2023-11-30T21:51:36.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.535+0000 7f1421c2f700 1 -- 172.21.15.49:0/1963584369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c0f5410 msgr2=0x7f141c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:36.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.535+0000 7f1421c2f700 1 --2- 172.21.15.49:0/1963584369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c0f5410 0x7f141c109430 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f140c013510 tx=0x7f140c00b040).stop 2023-11-30T21:51:36.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.535+0000 7f1421c2f700 1 -- 172.21.15.49:0/1963584369 shutdown_connections 2023-11-30T21:51:36.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.535+0000 7f1421c2f700 1 --2- 172.21.15.49:0/1963584369 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f140402ff60 0x7f1404032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:36.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.535+0000 7f1421c2f700 1 --2- 172.21.15.49:0/1963584369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f141c0f5410 0x7f141c109430 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:36.538 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.535+0000 7f1421c2f700 1 -- 172.21.15.49:0/1963584369 >> 172.21.15.49:0/1963584369 conn(0x7f141c0f0b70 msgr2=0x7f141c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:36.538 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.535+0000 7f1421c2f700 1 -- 172.21.15.49:0/1963584369 shutdown_connections 2023-11-30T21:51:36.538 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:36.535+0000 7f1421c2f700 1 -- 172.21.15.49:0/1963584369 wait complete. 2023-11-30T21:51:36.538 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:37.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:37 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:35.530584+0000 mgr.smithi027.brrjcl (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:37.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:37 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:36.534867+0000 mon.smithi027 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.49:0/1963584369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:37.931 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:37.932 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:39.195 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.192+0000 7f5ac2e58700 1 -- 172.21.15.49:0/3329557436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5abc0f6510 msgr2=0x7f5abc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:39.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.192+0000 7f5ac2e58700 1 --2- 172.21.15.49:0/3329557436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5abc0f6510 0x7f5abc0f6930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f5ab0009fb0 tx=0x7f5ab0009b30).stop 2023-11-30T21:51:39.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.192+0000 7f5ac2e58700 1 -- 172.21.15.49:0/3329557436 shutdown_connections 2023-11-30T21:51:39.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.192+0000 7f5ac2e58700 1 --2- 172.21.15.49:0/3329557436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5abc0f6510 0x7f5abc0f6930 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:39.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.192+0000 7f5ac2e58700 1 -- 172.21.15.49:0/3329557436 >> 172.21.15.49:0/3329557436 conn(0x7f5abc0f1c70 msgr2=0x7f5abc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:39.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.192+0000 7f5ac2e58700 1 -- 172.21.15.49:0/3329557436 shutdown_connections 2023-11-30T21:51:39.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.192+0000 7f5ac2e58700 1 -- 172.21.15.49:0/3329557436 wait complete. 2023-11-30T21:51:39.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.192+0000 7f5ac2e58700 1 Processor -- start 2023-11-30T21:51:39.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.192+0000 7f5ac2e58700 1 -- start start 2023-11-30T21:51:39.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.193+0000 7f5ac2e58700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5abc0f6510 0x7f5abc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:39.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.193+0000 7f5ac2e58700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc106c20 con 0x7f5abc0f6510 2023-11-30T21:51:39.196 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.193+0000 7f5ac0bf4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5abc0f6510 0x7f5abc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:39.197 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.193+0000 7f5ac0bf4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5abc0f6510 0x7f5abc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:46122/0 (socket says 172.21.15.49:46122) 2023-11-30T21:51:39.197 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.193+0000 7f5ac0bf4700 1 -- 172.21.15.49:0/1421670238 learned_addr learned my addr 172.21.15.49:0/1421670238 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:39.197 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.193+0000 7f5ac0bf4700 1 -- 172.21.15.49:0/1421670238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab0005040 con 0x7f5abc0f6510 2023-11-30T21:51:39.197 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.194+0000 7f5ac0bf4700 1 --2- 172.21.15.49:0/1421670238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5abc0f6510 0x7f5abc106730 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f5ab0013510 tx=0x7f5ab000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:39.197 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.194+0000 7f5ab97fa700 1 -- 172.21.15.49:0/1421670238 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ab0009320 con 0x7f5abc0f6510 2023-11-30T21:51:39.197 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.194+0000 7f5ac2e58700 1 -- 172.21.15.49:0/1421670238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5abc106e20 con 0x7f5abc0f6510 2023-11-30T21:51:39.197 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.194+0000 7f5ab97fa700 1 -- 172.21.15.49:0/1421670238 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ab0013f00 con 0x7f5abc0f6510 2023-11-30T21:51:39.197 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.194+0000 7f5ab97fa700 1 -- 172.21.15.49:0/1421670238 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ab0014d90 con 0x7f5abc0f6510 2023-11-30T21:51:39.197 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.194+0000 7f5ac2e58700 1 -- 172.21.15.49:0/1421670238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5abc1072b0 con 0x7f5abc0f6510 2023-11-30T21:51:39.198 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.195+0000 7f5ab97fa700 1 -- 172.21.15.49:0/1421670238 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ab0014420 con 0x7f5abc0f6510 2023-11-30T21:51:39.198 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.195+0000 7f5ac2e58700 1 -- 172.21.15.49:0/1421670238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5abc045ad0 con 0x7f5abc0f6510 2023-11-30T21:51:39.198 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.195+0000 7f5ab97fa700 1 --2- 172.21.15.49:0/1421670238 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5aac02ff60 0x7f5aac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:39.198 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.195+0000 7f5ab97fa700 1 -- 172.21.15.49:0/1421670238 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ab0040fb0 con 0x7f5abc0f6510 2023-11-30T21:51:39.198 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.195+0000 7f5abbfff700 1 --2- 172.21.15.49:0/1421670238 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5aac02ff60 0x7f5aac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:39.198 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.195+0000 7f5abbfff700 1 --2- 172.21.15.49:0/1421670238 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5aac02ff60 0x7f5aac032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f5aa80096a0 tx=0x7f5aa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:39.202 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.199+0000 7f5ab97fa700 1 -- 172.21.15.49:0/1421670238 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ab00095f0 con 0x7f5abc0f6510 2023-11-30T21:51:39.420 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.419+0000 7f5ac2e58700 1 -- 172.21.15.49:0/1421670238 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5abc0fbf60 con 0x7f5abc0f6510 2023-11-30T21:51:39.421 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.419+0000 7f5ab97fa700 1 -- 172.21.15.49:0/1421670238 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ab001db70 con 0x7f5abc0f6510 2023-11-30T21:51:39.421 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:39.421 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:39.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.421+0000 7f5ac2e58700 1 -- 172.21.15.49:0/1421670238 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5aac02ff60 msgr2=0x7f5aac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:39.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.421+0000 7f5ac2e58700 1 --2- 172.21.15.49:0/1421670238 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5aac02ff60 0x7f5aac032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f5aa80096a0 tx=0x7f5aa8006b40).stop 2023-11-30T21:51:39.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.421+0000 7f5ac2e58700 1 -- 172.21.15.49:0/1421670238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5abc0f6510 msgr2=0x7f5abc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:39.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.421+0000 7f5ac2e58700 1 --2- 172.21.15.49:0/1421670238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5abc0f6510 0x7f5abc106730 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f5ab0013510 tx=0x7f5ab000b040).stop 2023-11-30T21:51:39.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.421+0000 7f5ac2e58700 1 -- 172.21.15.49:0/1421670238 shutdown_connections 2023-11-30T21:51:39.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.421+0000 7f5ac2e58700 1 --2- 172.21.15.49:0/1421670238 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5aac02ff60 0x7f5aac032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:39.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.421+0000 7f5ac2e58700 1 --2- 172.21.15.49:0/1421670238 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5abc0f6510 0x7f5abc106730 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:39.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.421+0000 7f5ac2e58700 1 -- 172.21.15.49:0/1421670238 >> 172.21.15.49:0/1421670238 conn(0x7f5abc0f1c70 msgr2=0x7f5abc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:39.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.421+0000 7f5ac2e58700 1 -- 172.21.15.49:0/1421670238 shutdown_connections 2023-11-30T21:51:39.423 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:39.421+0000 7f5ac2e58700 1 -- 172.21.15.49:0/1421670238 wait complete. 2023-11-30T21:51:39.424 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:39.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:37.530844+0000 mgr.smithi027.brrjcl (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:40.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:40 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:39.421068+0000 mon.smithi027 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.49:0/1421670238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:40.799 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:40.800 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:41.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:41 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:39.531187+0000 mgr.smithi027.brrjcl (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:42.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.023+0000 7ff933e97700 1 -- 172.21.15.49:0/1755293909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0f6510 msgr2=0x7ff92c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:42.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.023+0000 7ff933e97700 1 --2- 172.21.15.49:0/1755293909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0f6510 0x7ff92c0f6930 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7ff928009fb0 tx=0x7ff928009b30).stop 2023-11-30T21:51:42.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.023+0000 7ff933e97700 1 -- 172.21.15.49:0/1755293909 shutdown_connections 2023-11-30T21:51:42.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.023+0000 7ff933e97700 1 --2- 172.21.15.49:0/1755293909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0f6510 0x7ff92c0f6930 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:42.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.023+0000 7ff933e97700 1 -- 172.21.15.49:0/1755293909 >> 172.21.15.49:0/1755293909 conn(0x7ff92c0f1c70 msgr2=0x7ff92c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:42.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.023+0000 7ff933e97700 1 -- 172.21.15.49:0/1755293909 shutdown_connections 2023-11-30T21:51:42.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.023+0000 7ff933e97700 1 -- 172.21.15.49:0/1755293909 wait complete. 2023-11-30T21:51:42.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.024+0000 7ff933e97700 1 Processor -- start 2023-11-30T21:51:42.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.024+0000 7ff933e97700 1 -- start start 2023-11-30T21:51:42.027 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.024+0000 7ff933e97700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0f6510 0x7ff92c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:42.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.024+0000 7ff933e97700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c106c20 con 0x7ff92c0f6510 2023-11-30T21:51:42.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.024+0000 7ff931c33700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0f6510 0x7ff92c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:42.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.024+0000 7ff931c33700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0f6510 0x7ff92c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:46144/0 (socket says 172.21.15.49:46144) 2023-11-30T21:51:42.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.024+0000 7ff931c33700 1 -- 172.21.15.49:0/2840786948 learned_addr learned my addr 172.21.15.49:0/2840786948 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:42.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.025+0000 7ff931c33700 1 -- 172.21.15.49:0/2840786948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff928005040 con 0x7ff92c0f6510 2023-11-30T21:51:42.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.025+0000 7ff931c33700 1 --2- 172.21.15.49:0/2840786948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0f6510 0x7ff92c106730 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7ff928014000 tx=0x7ff92800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:42.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.025+0000 7ff9227fc700 1 -- 172.21.15.49:0/2840786948 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9280095f0 con 0x7ff92c0f6510 2023-11-30T21:51:42.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.025+0000 7ff933e97700 1 -- 172.21.15.49:0/2840786948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92c106e20 con 0x7ff92c0f6510 2023-11-30T21:51:42.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.025+0000 7ff9227fc700 1 -- 172.21.15.49:0/2840786948 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff928007e70 con 0x7ff92c0f6510 2023-11-30T21:51:42.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.025+0000 7ff9227fc700 1 -- 172.21.15.49:0/2840786948 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff928014ea0 con 0x7ff92c0f6510 2023-11-30T21:51:42.028 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.025+0000 7ff933e97700 1 -- 172.21.15.49:0/2840786948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92c1072b0 con 0x7ff92c0f6510 2023-11-30T21:51:42.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.026+0000 7ff9227fc700 1 -- 172.21.15.49:0/2840786948 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff928014690 con 0x7ff92c0f6510 2023-11-30T21:51:42.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.026+0000 7ff933e97700 1 -- 172.21.15.49:0/2840786948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92c045ad0 con 0x7ff92c0f6510 2023-11-30T21:51:42.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.026+0000 7ff9227fc700 1 --2- 172.21.15.49:0/2840786948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff91802ff60 0x7ff918032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:42.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.026+0000 7ff9227fc700 1 -- 172.21.15.49:0/2840786948 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff928040870 con 0x7ff92c0f6510 2023-11-30T21:51:42.029 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.026+0000 7ff931432700 1 --2- 172.21.15.49:0/2840786948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff91802ff60 0x7ff918032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:42.030 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.027+0000 7ff931432700 1 --2- 172.21.15.49:0/2840786948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff91802ff60 0x7ff918032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ff91c0096a0 tx=0x7ff91c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:42.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.032+0000 7ff9227fc700 1 -- 172.21.15.49:0/2840786948 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff92800f300 con 0x7ff92c0f6510 2023-11-30T21:51:42.251 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.250+0000 7ff933e97700 1 -- 172.21.15.49:0/2840786948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff92c0fbf60 con 0x7ff92c0f6510 2023-11-30T21:51:42.252 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.250+0000 7ff9227fc700 1 -- 172.21.15.49:0/2840786948 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff92800f300 con 0x7ff92c0f6510 2023-11-30T21:51:42.252 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:42.252 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:42.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.252+0000 7ff933e97700 1 -- 172.21.15.49:0/2840786948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff91802ff60 msgr2=0x7ff918032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:42.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.252+0000 7ff933e97700 1 --2- 172.21.15.49:0/2840786948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff91802ff60 0x7ff918032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ff91c0096a0 tx=0x7ff91c006b40).stop 2023-11-30T21:51:42.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.252+0000 7ff933e97700 1 -- 172.21.15.49:0/2840786948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0f6510 msgr2=0x7ff92c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:42.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.252+0000 7ff933e97700 1 --2- 172.21.15.49:0/2840786948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0f6510 0x7ff92c106730 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7ff928014000 tx=0x7ff92800b040).stop 2023-11-30T21:51:42.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.252+0000 7ff933e97700 1 -- 172.21.15.49:0/2840786948 shutdown_connections 2023-11-30T21:51:42.254 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.252+0000 7ff933e97700 1 --2- 172.21.15.49:0/2840786948 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7ff91802ff60 0x7ff918032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:42.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.252+0000 7ff933e97700 1 --2- 172.21.15.49:0/2840786948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0f6510 0x7ff92c106730 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:42.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.252+0000 7ff933e97700 1 -- 172.21.15.49:0/2840786948 >> 172.21.15.49:0/2840786948 conn(0x7ff92c0f1c70 msgr2=0x7ff92c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:42.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.252+0000 7ff933e97700 1 -- 172.21.15.49:0/2840786948 shutdown_connections 2023-11-30T21:51:42.255 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:42.252+0000 7ff933e97700 1 -- 172.21.15.49:0/2840786948 wait complete. 2023-11-30T21:51:42.255 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:43.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:41.531536+0000 mgr.smithi027.brrjcl (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:43.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:42.252088+0000 mon.smithi027 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.49:0/2840786948' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:43.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:43.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:44.965 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.964+0000 7f02effff700 1 -- 172.21.15.49:0/1712914468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f00f51a0 msgr2=0x7f02f00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:44.965 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.964+0000 7f02effff700 1 --2- 172.21.15.49:0/1712914468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f00f51a0 0x7f02f00f55c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f02e0009fb0 tx=0x7f02e0009b30).stop 2023-11-30T21:51:44.965 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.964+0000 7f02effff700 1 -- 172.21.15.49:0/1712914468 shutdown_connections 2023-11-30T21:51:44.966 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.964+0000 7f02effff700 1 --2- 172.21.15.49:0/1712914468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f00f51a0 0x7f02f00f55c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:44.966 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.964+0000 7f02effff700 1 -- 172.21.15.49:0/1712914468 >> 172.21.15.49:0/1712914468 conn(0x7f02f00f09a0 msgr2=0x7f02f00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:44.966 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.964+0000 7f02effff700 1 -- 172.21.15.49:0/1712914468 shutdown_connections 2023-11-30T21:51:44.966 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.964+0000 7f02effff700 1 -- 172.21.15.49:0/1712914468 wait complete. 2023-11-30T21:51:44.966 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.964+0000 7f02effff700 1 Processor -- start 2023-11-30T21:51:44.966 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.964+0000 7f02effff700 1 -- start start 2023-11-30T21:51:44.966 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.965+0000 7f02effff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f00f51a0 0x7f02f01064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:44.966 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.965+0000 7f02effff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f0106990 con 0x7f02f00f51a0 2023-11-30T21:51:44.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.965+0000 7f02eeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f00f51a0 0x7f02f01064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:44.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.965+0000 7f02eeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f00f51a0 0x7f02f01064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:46162/0 (socket says 172.21.15.49:46162) 2023-11-30T21:51:44.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.965+0000 7f02eeffd700 1 -- 172.21.15.49:0/421844554 learned_addr learned my addr 172.21.15.49:0/421844554 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:44.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.965+0000 7f02eeffd700 1 -- 172.21.15.49:0/421844554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e0005040 con 0x7f02f00f51a0 2023-11-30T21:51:44.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.965+0000 7f02eeffd700 1 --2- 172.21.15.49:0/421844554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f00f51a0 0x7f02f01064a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f02e0013510 tx=0x7f02e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:44.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.966+0000 7f02d77fe700 1 -- 172.21.15.49:0/421844554 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02e0009320 con 0x7f02f00f51a0 2023-11-30T21:51:44.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.966+0000 7f02effff700 1 -- 172.21.15.49:0/421844554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f0106b90 con 0x7f02f00f51a0 2023-11-30T21:51:44.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.966+0000 7f02d77fe700 1 -- 172.21.15.49:0/421844554 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02e0013f00 con 0x7f02f00f51a0 2023-11-30T21:51:44.968 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.966+0000 7f02d77fe700 1 -- 172.21.15.49:0/421844554 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02e0014dc0 con 0x7f02f00f51a0 2023-11-30T21:51:44.969 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.966+0000 7f02effff700 1 -- 172.21.15.49:0/421844554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f0107020 con 0x7f02f00f51a0 2023-11-30T21:51:44.969 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.966+0000 7f02d77fe700 1 -- 172.21.15.49:0/421844554 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02e001d3d0 con 0x7f02f00f51a0 2023-11-30T21:51:44.969 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.966+0000 7f02effff700 1 -- 172.21.15.49:0/421844554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f0045ad0 con 0x7f02f00f51a0 2023-11-30T21:51:44.969 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.967+0000 7f02d77fe700 1 --2- 172.21.15.49:0/421844554 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f02d802ff60 0x7f02d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:44.969 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.967+0000 7f02d77fe700 1 -- 172.21.15.49:0/421844554 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02e0041330 con 0x7f02f00f51a0 2023-11-30T21:51:44.969 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.967+0000 7f02ee7fc700 1 --2- 172.21.15.49:0/421844554 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f02d802ff60 0x7f02d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:44.969 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.967+0000 7f02ee7fc700 1 --2- 172.21.15.49:0/421844554 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f02d802ff60 0x7f02d8032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f02e40096a0 tx=0x7f02e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:44.972 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:44.971+0000 7f02d77fe700 1 -- 172.21.15.49:0/421844554 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02e001cb10 con 0x7f02f00f51a0 2023-11-30T21:51:45.190 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:45.190+0000 7f02effff700 1 -- 172.21.15.49:0/421844554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02f0107ed0 con 0x7f02f00f51a0 2023-11-30T21:51:45.191 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:45.191+0000 7f02d77fe700 1 -- 172.21.15.49:0/421844554 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f02e001cb10 con 0x7f02f00f51a0 2023-11-30T21:51:45.191 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:45.192 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:45.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:45.192+0000 7f02effff700 1 -- 172.21.15.49:0/421844554 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f02d802ff60 msgr2=0x7f02d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:45.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:45.192+0000 7f02effff700 1 --2- 172.21.15.49:0/421844554 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f02d802ff60 0x7f02d8032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f02e40096a0 tx=0x7f02e4006b40).stop 2023-11-30T21:51:45.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:45.192+0000 7f02effff700 1 -- 172.21.15.49:0/421844554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f00f51a0 msgr2=0x7f02f01064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:45.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:45.192+0000 7f02effff700 1 --2- 172.21.15.49:0/421844554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f00f51a0 0x7f02f01064a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f02e0013510 tx=0x7f02e000b040).stop 2023-11-30T21:51:45.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:45.192+0000 7f02effff700 1 -- 172.21.15.49:0/421844554 shutdown_connections 2023-11-30T21:51:45.193 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:45.192+0000 7f02effff700 1 --2- 172.21.15.49:0/421844554 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f02d802ff60 0x7f02d8032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:45.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:45.192+0000 7f02effff700 1 --2- 172.21.15.49:0/421844554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02f00f51a0 0x7f02f01064a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:45.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:45.192+0000 7f02effff700 1 -- 172.21.15.49:0/421844554 >> 172.21.15.49:0/421844554 conn(0x7f02f00f09a0 msgr2=0x7f02f005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:45.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:45.193+0000 7f02effff700 1 -- 172.21.15.49:0/421844554 shutdown_connections 2023-11-30T21:51:45.194 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:45.193+0000 7f02effff700 1 -- 172.21.15.49:0/421844554 wait complete. 2023-11-30T21:51:45.194 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:45 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:43.531811+0000 mgr.smithi027.brrjcl (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:45 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:45.190964+0000 mon.smithi027 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.49:0/421844554' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:46.634 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:46.634 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:47.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:47 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:45.532118+0000 mgr.smithi027.brrjcl (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:47.892 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.890+0000 7fd198514700 1 -- 172.21.15.49:0/484145448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1900f5410 msgr2=0x7fd1900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:47.892 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.890+0000 7fd198514700 1 --2- 172.21.15.49:0/484145448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1900f5410 0x7fd1900f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fd180009fb0 tx=0x7fd180009b30).stop 2023-11-30T21:51:47.892 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.890+0000 7fd198514700 1 -- 172.21.15.49:0/484145448 shutdown_connections 2023-11-30T21:51:47.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.890+0000 7fd198514700 1 --2- 172.21.15.49:0/484145448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1900f5410 0x7fd1900f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:47.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.890+0000 7fd198514700 1 -- 172.21.15.49:0/484145448 >> 172.21.15.49:0/484145448 conn(0x7fd1900f0b70 msgr2=0x7fd1900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:47.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.890+0000 7fd198514700 1 -- 172.21.15.49:0/484145448 shutdown_connections 2023-11-30T21:51:47.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.890+0000 7fd198514700 1 -- 172.21.15.49:0/484145448 wait complete. 2023-11-30T21:51:47.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.891+0000 7fd198514700 1 Processor -- start 2023-11-30T21:51:47.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.891+0000 7fd198514700 1 -- start start 2023-11-30T21:51:47.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.891+0000 7fd198514700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1900f5410 0x7fd190108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:47.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.891+0000 7fd198514700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd190108e00 con 0x7fd1900f5410 2023-11-30T21:51:47.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.891+0000 7fd1962b0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1900f5410 0x7fd190108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:47.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.891+0000 7fd1962b0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1900f5410 0x7fd190108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:46172/0 (socket says 172.21.15.49:46172) 2023-11-30T21:51:47.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.891+0000 7fd1962b0700 1 -- 172.21.15.49:0/3011694464 learned_addr learned my addr 172.21.15.49:0/3011694464 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:47.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.891+0000 7fd1962b0700 1 -- 172.21.15.49:0/3011694464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd180005040 con 0x7fd1900f5410 2023-11-30T21:51:47.893 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.891+0000 7fd1962b0700 1 --2- 172.21.15.49:0/3011694464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1900f5410 0x7fd190108910 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fd180013510 tx=0x7fd18000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:47.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.892+0000 7fd186ffd700 1 -- 172.21.15.49:0/3011694464 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd180009320 con 0x7fd1900f5410 2023-11-30T21:51:47.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.892+0000 7fd186ffd700 1 -- 172.21.15.49:0/3011694464 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd180013f00 con 0x7fd1900f5410 2023-11-30T21:51:47.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.892+0000 7fd198514700 1 -- 172.21.15.49:0/3011694464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd190109000 con 0x7fd1900f5410 2023-11-30T21:51:47.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.892+0000 7fd186ffd700 1 -- 172.21.15.49:0/3011694464 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd180014d90 con 0x7fd1900f5410 2023-11-30T21:51:47.894 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.892+0000 7fd198514700 1 -- 172.21.15.49:0/3011694464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd190109490 con 0x7fd1900f5410 2023-11-30T21:51:47.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.892+0000 7fd186ffd700 1 -- 172.21.15.49:0/3011694464 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1800263a0 con 0x7fd1900f5410 2023-11-30T21:51:47.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.893+0000 7fd186ffd700 1 --2- 172.21.15.49:0/3011694464 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd17c02ff60 0x7fd17c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:47.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.893+0000 7fd198514700 1 -- 172.21.15.49:0/3011694464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd190045ad0 con 0x7fd1900f5410 2023-11-30T21:51:47.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.893+0000 7fd186ffd700 1 -- 172.21.15.49:0/3011694464 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd180041000 con 0x7fd1900f5410 2023-11-30T21:51:47.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.893+0000 7fd195aaf700 1 --2- 172.21.15.49:0/3011694464 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd17c02ff60 0x7fd17c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:47.895 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.893+0000 7fd195aaf700 1 --2- 172.21.15.49:0/3011694464 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd17c02ff60 0x7fd17c032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fd18c0096a0 tx=0x7fd18c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:47.898 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:47.897+0000 7fd186ffd700 1 -- 172.21.15.49:0/3011694464 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd180012920 con 0x7fd1900f5410 2023-11-30T21:51:48.114 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:48.113+0000 7fd198514700 1 -- 172.21.15.49:0/3011694464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd1900faf50 con 0x7fd1900f5410 2023-11-30T21:51:48.114 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:48.114+0000 7fd186ffd700 1 -- 172.21.15.49:0/3011694464 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd180012330 con 0x7fd1900f5410 2023-11-30T21:51:48.114 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:48.115 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:48.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:48.115+0000 7fd198514700 1 -- 172.21.15.49:0/3011694464 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd17c02ff60 msgr2=0x7fd17c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:48.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:48.115+0000 7fd198514700 1 --2- 172.21.15.49:0/3011694464 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd17c02ff60 0x7fd17c032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fd18c0096a0 tx=0x7fd18c006b40).stop 2023-11-30T21:51:48.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:48.115+0000 7fd198514700 1 -- 172.21.15.49:0/3011694464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1900f5410 msgr2=0x7fd190108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:48.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:48.115+0000 7fd198514700 1 --2- 172.21.15.49:0/3011694464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1900f5410 0x7fd190108910 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fd180013510 tx=0x7fd18000b040).stop 2023-11-30T21:51:48.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:48.115+0000 7fd198514700 1 -- 172.21.15.49:0/3011694464 shutdown_connections 2023-11-30T21:51:48.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:48.115+0000 7fd198514700 1 --2- 172.21.15.49:0/3011694464 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd17c02ff60 0x7fd17c032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:48.116 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:48.115+0000 7fd198514700 1 --2- 172.21.15.49:0/3011694464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1900f5410 0x7fd190108910 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:48.117 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:48.115+0000 7fd198514700 1 -- 172.21.15.49:0/3011694464 >> 172.21.15.49:0/3011694464 conn(0x7fd1900f0b70 msgr2=0x7fd19005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:48.117 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:48.115+0000 7fd198514700 1 -- 172.21.15.49:0/3011694464 shutdown_connections 2023-11-30T21:51:48.117 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:48.115+0000 7fd198514700 1 -- 172.21.15.49:0/3011694464 wait complete. 2023-11-30T21:51:48.117 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:48.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:48 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:48.114132+0000 mon.smithi027 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.49:0/3011694464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:49.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:49 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:47.532479+0000 mgr.smithi027.brrjcl (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:49.516 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:49.516 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:50.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.849+0000 7f1d1aa69700 1 -- 172.21.15.49:0/1802064575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d140f5410 msgr2=0x7f1d140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:50.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.849+0000 7f1d1aa69700 1 --2- 172.21.15.49:0/1802064575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d140f5410 0x7f1d140f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f1d08009fb0 tx=0x7f1d08009b30).stop 2023-11-30T21:51:50.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.849+0000 7f1d1aa69700 1 -- 172.21.15.49:0/1802064575 shutdown_connections 2023-11-30T21:51:50.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.849+0000 7f1d1aa69700 1 --2- 172.21.15.49:0/1802064575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d140f5410 0x7f1d140f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:50.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.849+0000 7f1d1aa69700 1 -- 172.21.15.49:0/1802064575 >> 172.21.15.49:0/1802064575 conn(0x7f1d140f0b70 msgr2=0x7f1d140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:50.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.849+0000 7f1d1aa69700 1 -- 172.21.15.49:0/1802064575 shutdown_connections 2023-11-30T21:51:50.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.849+0000 7f1d1aa69700 1 -- 172.21.15.49:0/1802064575 wait complete. 2023-11-30T21:51:50.854 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.850+0000 7f1d1aa69700 1 Processor -- start 2023-11-30T21:51:50.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.850+0000 7f1d1aa69700 1 -- start start 2023-11-30T21:51:50.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.850+0000 7f1d1aa69700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d140f5410 0x7f1d14106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:50.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.850+0000 7f1d1aa69700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d14106c00 con 0x7f1d140f5410 2023-11-30T21:51:50.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.851+0000 7f1d18805700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d140f5410 0x7f1d14106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:50.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.851+0000 7f1d18805700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d140f5410 0x7f1d14106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:56672/0 (socket says 172.21.15.49:56672) 2023-11-30T21:51:50.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.851+0000 7f1d18805700 1 -- 172.21.15.49:0/3215796798 learned_addr learned my addr 172.21.15.49:0/3215796798 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:50.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.851+0000 7f1d18805700 1 -- 172.21.15.49:0/3215796798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d08005040 con 0x7f1d140f5410 2023-11-30T21:51:50.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.851+0000 7f1d18805700 1 --2- 172.21.15.49:0/3215796798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d140f5410 0x7f1d14106710 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f1d08014000 tx=0x7f1d0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:50.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.851+0000 7f1d117fa700 1 -- 172.21.15.49:0/3215796798 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d080095f0 con 0x7f1d140f5410 2023-11-30T21:51:50.855 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.851+0000 7f1d1aa69700 1 -- 172.21.15.49:0/3215796798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d14106e00 con 0x7f1d140f5410 2023-11-30T21:51:50.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.851+0000 7f1d117fa700 1 -- 172.21.15.49:0/3215796798 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d08007e70 con 0x7f1d140f5410 2023-11-30T21:51:50.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.851+0000 7f1d117fa700 1 -- 172.21.15.49:0/3215796798 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d08014ea0 con 0x7f1d140f5410 2023-11-30T21:51:50.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.851+0000 7f1d1aa69700 1 -- 172.21.15.49:0/3215796798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d14107290 con 0x7f1d140f5410 2023-11-30T21:51:50.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.852+0000 7f1d117fa700 1 -- 172.21.15.49:0/3215796798 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d080079c0 con 0x7f1d140f5410 2023-11-30T21:51:50.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.852+0000 7f1d1aa69700 1 -- 172.21.15.49:0/3215796798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d14045ad0 con 0x7f1d140f5410 2023-11-30T21:51:50.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.852+0000 7f1d117fa700 1 --2- 172.21.15.49:0/3215796798 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f1d0402ffb0 0x7f1d04032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:50.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.852+0000 7f1d117fa700 1 -- 172.21.15.49:0/3215796798 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d08040820 con 0x7f1d140f5410 2023-11-30T21:51:50.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.853+0000 7f1d13fff700 1 --2- 172.21.15.49:0/3215796798 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f1d0402ffb0 0x7f1d04032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:50.856 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.853+0000 7f1d13fff700 1 --2- 172.21.15.49:0/3215796798 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f1d0402ffb0 0x7f1d04032470 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f1d000096a0 tx=0x7f1d00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:50.859 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:50.857+0000 7f1d117fa700 1 -- 172.21.15.49:0/3215796798 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d0800f950 con 0x7f1d140f5410 2023-11-30T21:51:51.077 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:51.077+0000 7f1d1aa69700 1 -- 172.21.15.49:0/3215796798 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d140fae60 con 0x7f1d140f5410 2023-11-30T21:51:51.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:51.077+0000 7f1d117fa700 1 -- 172.21.15.49:0/3215796798 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1d08010230 con 0x7f1d140f5410 2023-11-30T21:51:51.078 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:51.079 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:51.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:51.079+0000 7f1d1aa69700 1 -- 172.21.15.49:0/3215796798 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f1d0402ffb0 msgr2=0x7f1d04032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:51.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:51.079+0000 7f1d1aa69700 1 --2- 172.21.15.49:0/3215796798 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f1d0402ffb0 0x7f1d04032470 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f1d000096a0 tx=0x7f1d00006b40).stop 2023-11-30T21:51:51.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:51.079+0000 7f1d1aa69700 1 -- 172.21.15.49:0/3215796798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d140f5410 msgr2=0x7f1d14106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:51.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:51.079+0000 7f1d1aa69700 1 --2- 172.21.15.49:0/3215796798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d140f5410 0x7f1d14106710 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f1d08014000 tx=0x7f1d0800b040).stop 2023-11-30T21:51:51.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:51.079+0000 7f1d1aa69700 1 -- 172.21.15.49:0/3215796798 shutdown_connections 2023-11-30T21:51:51.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:51.079+0000 7f1d1aa69700 1 --2- 172.21.15.49:0/3215796798 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f1d0402ffb0 0x7f1d04032470 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:51.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:51.079+0000 7f1d1aa69700 1 --2- 172.21.15.49:0/3215796798 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d140f5410 0x7f1d14106710 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:51.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:51.079+0000 7f1d1aa69700 1 -- 172.21.15.49:0/3215796798 >> 172.21.15.49:0/3215796798 conn(0x7f1d140f0b70 msgr2=0x7f1d1405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:51.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:51.079+0000 7f1d1aa69700 1 -- 172.21.15.49:0/3215796798 shutdown_connections 2023-11-30T21:51:51.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:51.079+0000 7f1d1aa69700 1 -- 172.21.15.49:0/3215796798 wait complete. 2023-11-30T21:51:51.081 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:51.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:49.532781+0000 mgr.smithi027.brrjcl (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:51.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:51.078077+0000 mon.smithi027 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.49:0/3215796798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:52.469 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:52.469 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:53.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:53 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:51.533064+0000 mgr.smithi027.brrjcl (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:53.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.687+0000 7f56a617e700 1 -- 172.21.15.49:0/2783762756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a00f51a0 msgr2=0x7f56a00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:53.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.687+0000 7f56a617e700 1 --2- 172.21.15.49:0/2783762756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a00f51a0 0x7f56a00f55c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f5694009fb0 tx=0x7f5694009b30).stop 2023-11-30T21:51:53.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.687+0000 7f56a617e700 1 -- 172.21.15.49:0/2783762756 shutdown_connections 2023-11-30T21:51:53.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.687+0000 7f56a617e700 1 --2- 172.21.15.49:0/2783762756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a00f51a0 0x7f56a00f55c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:53.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.687+0000 7f56a617e700 1 -- 172.21.15.49:0/2783762756 >> 172.21.15.49:0/2783762756 conn(0x7f56a00f09a0 msgr2=0x7f56a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:53.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.687+0000 7f56a617e700 1 -- 172.21.15.49:0/2783762756 shutdown_connections 2023-11-30T21:51:53.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.687+0000 7f56a617e700 1 -- 172.21.15.49:0/2783762756 wait complete. 2023-11-30T21:51:53.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.688+0000 7f56a617e700 1 Processor -- start 2023-11-30T21:51:53.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.688+0000 7f56a617e700 1 -- start start 2023-11-30T21:51:53.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.688+0000 7f56a617e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a00f51a0 0x7f56a01064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:53.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.688+0000 7f56a617e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a0106990 con 0x7f56a00f51a0 2023-11-30T21:51:53.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.688+0000 7f56a517c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a00f51a0 0x7f56a01064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:53.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.688+0000 7f56a517c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a00f51a0 0x7f56a01064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:56690/0 (socket says 172.21.15.49:56690) 2023-11-30T21:51:53.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.688+0000 7f56a517c700 1 -- 172.21.15.49:0/2398891909 learned_addr learned my addr 172.21.15.49:0/2398891909 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:53.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.688+0000 7f56a517c700 1 -- 172.21.15.49:0/2398891909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5694005040 con 0x7f56a00f51a0 2023-11-30T21:51:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.688+0000 7f56a517c700 1 --2- 172.21.15.49:0/2398891909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a00f51a0 0x7f56a01064a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f5694013510 tx=0x7f569400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.689+0000 7f5691ffb700 1 -- 172.21.15.49:0/2398891909 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5694009320 con 0x7f56a00f51a0 2023-11-30T21:51:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.689+0000 7f56a617e700 1 -- 172.21.15.49:0/2398891909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a0106b90 con 0x7f56a00f51a0 2023-11-30T21:51:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.689+0000 7f56a617e700 1 -- 172.21.15.49:0/2398891909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a0107020 con 0x7f56a00f51a0 2023-11-30T21:51:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.689+0000 7f5691ffb700 1 -- 172.21.15.49:0/2398891909 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5694013f00 con 0x7f56a00f51a0 2023-11-30T21:51:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.689+0000 7f5691ffb700 1 -- 172.21.15.49:0/2398891909 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5694014d90 con 0x7f56a00f51a0 2023-11-30T21:51:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.690+0000 7f5691ffb700 1 -- 172.21.15.49:0/2398891909 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56940263a0 con 0x7f56a00f51a0 2023-11-30T21:51:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.690+0000 7f56a617e700 1 -- 172.21.15.49:0/2398891909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5680004fa0 con 0x7f56a00f51a0 2023-11-30T21:51:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.690+0000 7f5691ffb700 1 --2- 172.21.15.49:0/2398891909 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f568c02ff60 0x7f568c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.690+0000 7f5691ffb700 1 -- 172.21.15.49:0/2398891909 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56940412a0 con 0x7f56a00f51a0 2023-11-30T21:51:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.690+0000 7f56a497b700 1 --2- 172.21.15.49:0/2398891909 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f568c02ff60 0x7f568c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.691+0000 7f56a497b700 1 --2- 172.21.15.49:0/2398891909 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f568c02ff60 0x7f568c032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f569c0096a0 tx=0x7f569c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:53.695 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.695+0000 7f5691ffb700 1 -- 172.21.15.49:0/2398891909 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f569401d700 con 0x7f56a00f51a0 2023-11-30T21:51:53.913 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.912+0000 7f56a617e700 1 -- 172.21.15.49:0/2398891909 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5680005670 con 0x7f56a00f51a0 2023-11-30T21:51:53.913 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.913+0000 7f5691ffb700 1 -- 172.21.15.49:0/2398891909 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5694012330 con 0x7f56a00f51a0 2023-11-30T21:51:53.914 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:53.914 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:53.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.914+0000 7f56a617e700 1 -- 172.21.15.49:0/2398891909 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f568c02ff60 msgr2=0x7f568c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:53.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.914+0000 7f56a617e700 1 --2- 172.21.15.49:0/2398891909 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f568c02ff60 0x7f568c032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f569c0096a0 tx=0x7f569c006b40).stop 2023-11-30T21:51:53.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.914+0000 7f56a617e700 1 -- 172.21.15.49:0/2398891909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a00f51a0 msgr2=0x7f56a01064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:53.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.914+0000 7f56a617e700 1 --2- 172.21.15.49:0/2398891909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a00f51a0 0x7f56a01064a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f5694013510 tx=0x7f569400b040).stop 2023-11-30T21:51:53.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.914+0000 7f56a617e700 1 -- 172.21.15.49:0/2398891909 shutdown_connections 2023-11-30T21:51:53.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.914+0000 7f56a617e700 1 --2- 172.21.15.49:0/2398891909 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f568c02ff60 0x7f568c032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:53.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.914+0000 7f56a617e700 1 --2- 172.21.15.49:0/2398891909 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a00f51a0 0x7f56a01064a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:53.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.914+0000 7f56a617e700 1 -- 172.21.15.49:0/2398891909 >> 172.21.15.49:0/2398891909 conn(0x7f56a00f09a0 msgr2=0x7f56a005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:53.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.914+0000 7f56a617e700 1 -- 172.21.15.49:0/2398891909 shutdown_connections 2023-11-30T21:51:53.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:53.914+0000 7f56a617e700 1 -- 172.21.15.49:0/2398891909 wait complete. 2023-11-30T21:51:53.918 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:54.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:54 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:53.913482+0000 mon.smithi027 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.49:0/2398891909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:55.313 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:55.313 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:55.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:53.533353+0000 mgr.smithi027.brrjcl (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:56.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.490+0000 7f8adafd0700 1 -- 172.21.15.49:0/4112487829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad4060220 msgr2=0x7f8ad40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:56.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.490+0000 7f8adafd0700 1 --2- 172.21.15.49:0/4112487829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad4060220 0x7f8ad40f9150 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f8ac8009fb0 tx=0x7f8ac8009b30).stop 2023-11-30T21:51:56.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.490+0000 7f8adafd0700 1 -- 172.21.15.49:0/4112487829 shutdown_connections 2023-11-30T21:51:56.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.490+0000 7f8adafd0700 1 --2- 172.21.15.49:0/4112487829 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad4060220 0x7f8ad40f9150 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:56.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.491+0000 7f8adafd0700 1 -- 172.21.15.49:0/4112487829 >> 172.21.15.49:0/4112487829 conn(0x7f8ad40f0b70 msgr2=0x7f8ad40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:56.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.491+0000 7f8adafd0700 1 -- 172.21.15.49:0/4112487829 shutdown_connections 2023-11-30T21:51:56.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.491+0000 7f8adafd0700 1 -- 172.21.15.49:0/4112487829 wait complete. 2023-11-30T21:51:56.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.491+0000 7f8adafd0700 1 Processor -- start 2023-11-30T21:51:56.493 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.491+0000 7f8adafd0700 1 -- start start 2023-11-30T21:51:56.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.491+0000 7f8adafd0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad40f6b90 0x7f8ad40f5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:56.494 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.491+0000 7f8adafd0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad40f6fb0 con 0x7f8ad40f6b90 2023-11-30T21:51:56.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.492+0000 7f8ad8d6c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad40f6b90 0x7f8ad40f5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:56.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.492+0000 7f8ad8d6c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad40f6b90 0x7f8ad40f5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:56708/0 (socket says 172.21.15.49:56708) 2023-11-30T21:51:56.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.492+0000 7f8ad8d6c700 1 -- 172.21.15.49:0/2072348689 learned_addr learned my addr 172.21.15.49:0/2072348689 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:56.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.492+0000 7f8ad8d6c700 1 -- 172.21.15.49:0/2072348689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac8005040 con 0x7f8ad40f6b90 2023-11-30T21:51:56.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.492+0000 7f8ad8d6c700 1 --2- 172.21.15.49:0/2072348689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad40f6b90 0x7f8ad40f5050 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f8ac8009f80 tx=0x7f8ac8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:56.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.492+0000 7f8ad17fa700 1 -- 172.21.15.49:0/2072348689 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ac80136c0 con 0x7f8ad40f6b90 2023-11-30T21:51:56.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.492+0000 7f8adafd0700 1 -- 172.21.15.49:0/2072348689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad40f5540 con 0x7f8ad40f6b90 2023-11-30T21:51:56.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.492+0000 7f8ad17fa700 1 -- 172.21.15.49:0/2072348689 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ac80095f0 con 0x7f8ad40f6b90 2023-11-30T21:51:56.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.492+0000 7f8ad17fa700 1 -- 172.21.15.49:0/2072348689 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ac8014420 con 0x7f8ad40f6b90 2023-11-30T21:51:56.495 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.493+0000 7f8adafd0700 1 -- 172.21.15.49:0/2072348689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad40f5a30 con 0x7f8ad40f6b90 2023-11-30T21:51:56.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.493+0000 7f8ad17fa700 1 -- 172.21.15.49:0/2072348689 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ac8025060 con 0x7f8ad40f6b90 2023-11-30T21:51:56.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.493+0000 7f8adafd0700 1 -- 172.21.15.49:0/2072348689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad4045ad0 con 0x7f8ad40f6b90 2023-11-30T21:51:56.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.494+0000 7f8ad17fa700 1 --2- 172.21.15.49:0/2072348689 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8ac402ff60 0x7f8ac4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:56.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.494+0000 7f8ad17fa700 1 -- 172.21.15.49:0/2072348689 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ac8040db0 con 0x7f8ad40f6b90 2023-11-30T21:51:56.496 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.494+0000 7f8ad3fff700 1 --2- 172.21.15.49:0/2072348689 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8ac402ff60 0x7f8ac4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:56.497 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.494+0000 7f8ad3fff700 1 --2- 172.21.15.49:0/2072348689 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8ac402ff60 0x7f8ac4032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f8ac00096a0 tx=0x7f8ac0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:56.500 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.498+0000 7f8ad17fa700 1 -- 172.21.15.49:0/2072348689 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ac8004490 con 0x7f8ad40f6b90 2023-11-30T21:51:56.718 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.717+0000 7f8adafd0700 1 -- 172.21.15.49:0/2072348689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ad410c990 con 0x7f8ad40f6b90 2023-11-30T21:51:56.719 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.717+0000 7f8ad17fa700 1 -- 172.21.15.49:0/2072348689 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8ac802d2d0 con 0x7f8ad40f6b90 2023-11-30T21:51:56.719 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:56.719 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:56.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.719+0000 7f8adafd0700 1 -- 172.21.15.49:0/2072348689 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8ac402ff60 msgr2=0x7f8ac4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:56.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.719+0000 7f8adafd0700 1 --2- 172.21.15.49:0/2072348689 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8ac402ff60 0x7f8ac4032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f8ac00096a0 tx=0x7f8ac0006b40).stop 2023-11-30T21:51:56.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.719+0000 7f8adafd0700 1 -- 172.21.15.49:0/2072348689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad40f6b90 msgr2=0x7f8ad40f5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:56.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.719+0000 7f8adafd0700 1 --2- 172.21.15.49:0/2072348689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad40f6b90 0x7f8ad40f5050 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f8ac8009f80 tx=0x7f8ac8009b30).stop 2023-11-30T21:51:56.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.719+0000 7f8adafd0700 1 -- 172.21.15.49:0/2072348689 shutdown_connections 2023-11-30T21:51:56.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.719+0000 7f8adafd0700 1 --2- 172.21.15.49:0/2072348689 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8ac402ff60 0x7f8ac4032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:56.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.719+0000 7f8adafd0700 1 --2- 172.21.15.49:0/2072348689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad40f6b90 0x7f8ad40f5050 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:56.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.719+0000 7f8adafd0700 1 -- 172.21.15.49:0/2072348689 >> 172.21.15.49:0/2072348689 conn(0x7f8ad40f0b70 msgr2=0x7f8ad40f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:56.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.719+0000 7f8adafd0700 1 -- 172.21.15.49:0/2072348689 shutdown_connections 2023-11-30T21:51:56.722 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:56.719+0000 7f8adafd0700 1 -- 172.21.15.49:0/2072348689 wait complete. 2023-11-30T21:51:56.722 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:51:57.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:57 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:55.533662+0000 mgr.smithi027.brrjcl (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:57.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:57 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:56.718958+0000 mon.smithi027 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.49:0/2072348689' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:51:58.135 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:51:58.135 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:51:59.403 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.400+0000 7fe95cdf6700 1 -- 172.21.15.49:0/4132256808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9580f47c0 msgr2=0x7fe9580f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:59.404 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.400+0000 7fe95cdf6700 1 --2- 172.21.15.49:0/4132256808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9580f47c0 0x7fe9580f4be0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fe948009960 tx=0x7fe948004d10).stop 2023-11-30T21:51:59.404 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.400+0000 7fe95cdf6700 1 -- 172.21.15.49:0/4132256808 shutdown_connections 2023-11-30T21:51:59.404 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.400+0000 7fe95cdf6700 1 --2- 172.21.15.49:0/4132256808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9580f47c0 0x7fe9580f4be0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:59.404 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.400+0000 7fe95cdf6700 1 -- 172.21.15.49:0/4132256808 >> 172.21.15.49:0/4132256808 conn(0x7fe9580f02e0 msgr2=0x7fe9580f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:59.404 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.400+0000 7fe95cdf6700 1 -- 172.21.15.49:0/4132256808 shutdown_connections 2023-11-30T21:51:59.404 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.400+0000 7fe95cdf6700 1 -- 172.21.15.49:0/4132256808 wait complete. 2023-11-30T21:51:59.404 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.400+0000 7fe95cdf6700 1 Processor -- start 2023-11-30T21:51:59.404 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.400+0000 7fe95cdf6700 1 -- start start 2023-11-30T21:51:59.404 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.400+0000 7fe95cdf6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9580f47c0 0x7fe958106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:59.404 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.400+0000 7fe95cdf6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe958106bf0 con 0x7fe9580f47c0 2023-11-30T21:51:59.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.401+0000 7fe95659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9580f47c0 0x7fe958106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:59.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.401+0000 7fe95659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9580f47c0 0x7fe958106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:58810/0 (socket says 172.21.15.49:58810) 2023-11-30T21:51:59.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.401+0000 7fe95659c700 1 -- 172.21.15.49:0/1099989021 learned_addr learned my addr 172.21.15.49:0/1099989021 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:51:59.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.401+0000 7fe95659c700 1 -- 172.21.15.49:0/1099989021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe948009640 con 0x7fe9580f47c0 2023-11-30T21:51:59.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.401+0000 7fe95659c700 1 --2- 172.21.15.49:0/1099989021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9580f47c0 0x7fe958106700 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fe948004b70 tx=0x7fe948004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:59.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.402+0000 7fe946ffd700 1 -- 172.21.15.49:0/1099989021 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe948018070 con 0x7fe9580f47c0 2023-11-30T21:51:59.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.402+0000 7fe946ffd700 1 -- 172.21.15.49:0/1099989021 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe94800b870 con 0x7fe9580f47c0 2023-11-30T21:51:59.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.402+0000 7fe95cdf6700 1 -- 172.21.15.49:0/1099989021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe958106df0 con 0x7fe9580f47c0 2023-11-30T21:51:59.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.402+0000 7fe946ffd700 1 -- 172.21.15.49:0/1099989021 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe94801c560 con 0x7fe9580f47c0 2023-11-30T21:51:59.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.402+0000 7fe95cdf6700 1 -- 172.21.15.49:0/1099989021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe958107280 con 0x7fe9580f47c0 2023-11-30T21:51:59.406 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.402+0000 7fe95cdf6700 1 -- 172.21.15.49:0/1099989021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe958045ad0 con 0x7fe9580f47c0 2023-11-30T21:51:59.407 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.402+0000 7fe946ffd700 1 -- 172.21.15.49:0/1099989021 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe948013630 con 0x7fe9580f47c0 2023-11-30T21:51:59.407 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.403+0000 7fe946ffd700 1 --2- 172.21.15.49:0/1099989021 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe94002ffb0 0x7fe940032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:51:59.407 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.403+0000 7fe946ffd700 1 -- 172.21.15.49:0/1099989021 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9480411b0 con 0x7fe9580f47c0 2023-11-30T21:51:59.407 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.403+0000 7fe955d9b700 1 --2- 172.21.15.49:0/1099989021 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe94002ffb0 0x7fe940032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:51:59.407 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.403+0000 7fe955d9b700 1 --2- 172.21.15.49:0/1099989021 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe94002ffb0 0x7fe940032470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fe94c0096a0 tx=0x7fe94c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:51:59.409 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.408+0000 7fe946ffd700 1 -- 172.21.15.49:0/1099989021 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe948013a30 con 0x7fe9580f47c0 2023-11-30T21:51:59.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:51:59 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:57.534029+0000 mgr.smithi027.brrjcl (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:51:59.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.625+0000 7fe95cdf6700 1 -- 172.21.15.49:0/1099989021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe958108130 con 0x7fe9580f47c0 2023-11-30T21:51:59.628 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.626+0000 7fe946ffd700 1 -- 172.21.15.49:0/1099989021 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe948011b10 con 0x7fe9580f47c0 2023-11-30T21:51:59.628 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:51:59.628 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:51:59.630 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.628+0000 7fe95cdf6700 1 -- 172.21.15.49:0/1099989021 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe94002ffb0 msgr2=0x7fe940032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:59.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.628+0000 7fe95cdf6700 1 --2- 172.21.15.49:0/1099989021 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe94002ffb0 0x7fe940032470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fe94c0096a0 tx=0x7fe94c006b40).stop 2023-11-30T21:51:59.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.628+0000 7fe95cdf6700 1 -- 172.21.15.49:0/1099989021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9580f47c0 msgr2=0x7fe958106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:51:59.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.628+0000 7fe95cdf6700 1 --2- 172.21.15.49:0/1099989021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9580f47c0 0x7fe958106700 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fe948004b70 tx=0x7fe948004d10).stop 2023-11-30T21:51:59.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.628+0000 7fe95cdf6700 1 -- 172.21.15.49:0/1099989021 shutdown_connections 2023-11-30T21:51:59.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.628+0000 7fe95cdf6700 1 --2- 172.21.15.49:0/1099989021 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe94002ffb0 0x7fe940032470 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:59.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.628+0000 7fe95cdf6700 1 --2- 172.21.15.49:0/1099989021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9580f47c0 0x7fe958106700 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:51:59.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.628+0000 7fe95cdf6700 1 -- 172.21.15.49:0/1099989021 >> 172.21.15.49:0/1099989021 conn(0x7fe9580f02e0 msgr2=0x7fe9580f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:51:59.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.628+0000 7fe95cdf6700 1 -- 172.21.15.49:0/1099989021 shutdown_connections 2023-11-30T21:51:59.631 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:51:59.628+0000 7fe95cdf6700 1 -- 172.21.15.49:0/1099989021 wait complete. 2023-11-30T21:51:59.631 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:00.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:00 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:51:59.627773+0000 mon.smithi027 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.49:0/1099989021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:01.047 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:01.048 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:01.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:01 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:51:59.534306+0000 mgr.smithi027.brrjcl (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:02.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.280+0000 7f051f487700 1 -- 172.21.15.49:0/3728643588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05180f5410 msgr2=0x7f05180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.280+0000 7f051f487700 1 --2- 172.21.15.49:0/3728643588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05180f5410 0x7f05180f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f0514009fb0 tx=0x7f0514009b30).stop 2023-11-30T21:52:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.281+0000 7f051f487700 1 -- 172.21.15.49:0/3728643588 shutdown_connections 2023-11-30T21:52:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.281+0000 7f051f487700 1 --2- 172.21.15.49:0/3728643588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05180f5410 0x7f05180f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.281+0000 7f051f487700 1 -- 172.21.15.49:0/3728643588 >> 172.21.15.49:0/3728643588 conn(0x7f05180f0b70 msgr2=0x7f05180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.281+0000 7f051f487700 1 -- 172.21.15.49:0/3728643588 shutdown_connections 2023-11-30T21:52:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.281+0000 7f051f487700 1 -- 172.21.15.49:0/3728643588 wait complete. 2023-11-30T21:52:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.281+0000 7f051f487700 1 Processor -- start 2023-11-30T21:52:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.281+0000 7f051f487700 1 -- start start 2023-11-30T21:52:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.281+0000 7f051f487700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05180f5410 0x7f05181093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.281+0000 7f051f487700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0518060220 con 0x7f05180f5410 2023-11-30T21:52:02.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.282+0000 7f051d223700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05180f5410 0x7f05181093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:02.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.282+0000 7f051d223700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05180f5410 0x7f05181093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:58832/0 (socket says 172.21.15.49:58832) 2023-11-30T21:52:02.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.282+0000 7f051d223700 1 -- 172.21.15.49:0/2994384123 learned_addr learned my addr 172.21.15.49:0/2994384123 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:02.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.282+0000 7f051d223700 1 -- 172.21.15.49:0/2994384123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0514005040 con 0x7f05180f5410 2023-11-30T21:52:02.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.282+0000 7f051d223700 1 --2- 172.21.15.49:0/2994384123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05180f5410 0x7f05181093f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f0514013510 tx=0x7f051400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:02.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.282+0000 7f050dffb700 1 -- 172.21.15.49:0/2994384123 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0514009320 con 0x7f05180f5410 2023-11-30T21:52:02.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.282+0000 7f051f487700 1 -- 172.21.15.49:0/2994384123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0518060420 con 0x7f05180f5410 2023-11-30T21:52:02.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.282+0000 7f050dffb700 1 -- 172.21.15.49:0/2994384123 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0514013f00 con 0x7f05180f5410 2023-11-30T21:52:02.286 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.282+0000 7f050dffb700 1 -- 172.21.15.49:0/2994384123 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0514014d90 con 0x7f05180f5410 2023-11-30T21:52:02.287 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.283+0000 7f051f487700 1 -- 172.21.15.49:0/2994384123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0518109aa0 con 0x7f05180f5410 2023-11-30T21:52:02.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.283+0000 7f050dffb700 1 -- 172.21.15.49:0/2994384123 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0514014420 con 0x7f05180f5410 2023-11-30T21:52:02.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.283+0000 7f051f487700 1 -- 172.21.15.49:0/2994384123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0518059100 con 0x7f05180f5410 2023-11-30T21:52:02.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.283+0000 7f050dffb700 1 --2- 172.21.15.49:0/2994384123 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f050402ff60 0x7f0504032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:02.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.284+0000 7f050dffb700 1 -- 172.21.15.49:0/2994384123 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0514041000 con 0x7f05180f5410 2023-11-30T21:52:02.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.284+0000 7f051ca22700 1 --2- 172.21.15.49:0/2994384123 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f050402ff60 0x7f0504032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:02.288 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.284+0000 7f051ca22700 1 --2- 172.21.15.49:0/2994384123 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f050402ff60 0x7f0504032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f05080096a0 tx=0x7f0508006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:02.290 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.288+0000 7f050dffb700 1 -- 172.21.15.49:0/2994384123 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05140095f0 con 0x7f05180f5410 2023-11-30T21:52:02.508 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.506+0000 7f051f487700 1 -- 172.21.15.49:0/2994384123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0518045ad0 con 0x7f05180f5410 2023-11-30T21:52:02.508 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.507+0000 7f050dffb700 1 -- 172.21.15.49:0/2994384123 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f051401db70 con 0x7f05180f5410 2023-11-30T21:52:02.509 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:02.509 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:02.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.508+0000 7f051f487700 1 -- 172.21.15.49:0/2994384123 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f050402ff60 msgr2=0x7f0504032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:02.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.508+0000 7f051f487700 1 --2- 172.21.15.49:0/2994384123 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f050402ff60 0x7f0504032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f05080096a0 tx=0x7f0508006b40).stop 2023-11-30T21:52:02.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.508+0000 7f051f487700 1 -- 172.21.15.49:0/2994384123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05180f5410 msgr2=0x7f05181093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:02.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.508+0000 7f051f487700 1 --2- 172.21.15.49:0/2994384123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05180f5410 0x7f05181093f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f0514013510 tx=0x7f051400b040).stop 2023-11-30T21:52:02.510 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.508+0000 7f051f487700 1 -- 172.21.15.49:0/2994384123 shutdown_connections 2023-11-30T21:52:02.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.509+0000 7f051f487700 1 --2- 172.21.15.49:0/2994384123 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f050402ff60 0x7f0504032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:02.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.509+0000 7f051f487700 1 --2- 172.21.15.49:0/2994384123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05180f5410 0x7f05181093f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:02.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.509+0000 7f051f487700 1 -- 172.21.15.49:0/2994384123 >> 172.21.15.49:0/2994384123 conn(0x7f05180f0b70 msgr2=0x7f05180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:02.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.509+0000 7f051f487700 1 -- 172.21.15.49:0/2994384123 shutdown_connections 2023-11-30T21:52:02.511 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:02.509+0000 7f051f487700 1 -- 172.21.15.49:0/2994384123 wait complete. 2023-11-30T21:52:02.511 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:03.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:03 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:01.534597+0000 mgr.smithi027.brrjcl (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:03.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:03 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:02.508365+0000 mon.smithi027 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.49:0/2994384123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:03.905 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:03.905 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:04.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:03.592584+0000 mon.smithi027 (mon.0) 292 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/mirror_snapshot_schedule"}]: dispatch 2023-11-30T21:52:04.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:03.598079+0000 mon.smithi027 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/trash_purge_schedule"}]: dispatch 2023-11-30T21:52:05.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.204+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/3324150669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080f5410 msgr2=0x7f5d080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:05.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.204+0000 7f5d0d2bb700 1 --2- 172.21.15.49:0/3324150669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080f5410 0x7f5d080f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f5cfc009fb0 tx=0x7f5cfc009b30).stop 2023-11-30T21:52:05.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.204+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/3324150669 shutdown_connections 2023-11-30T21:52:05.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.204+0000 7f5d0d2bb700 1 --2- 172.21.15.49:0/3324150669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080f5410 0x7f5d080f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:05.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.204+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/3324150669 >> 172.21.15.49:0/3324150669 conn(0x7f5d080f0b70 msgr2=0x7f5d080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:05.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.204+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/3324150669 shutdown_connections 2023-11-30T21:52:05.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.204+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/3324150669 wait complete. 2023-11-30T21:52:05.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.205+0000 7f5d0d2bb700 1 Processor -- start 2023-11-30T21:52:05.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.205+0000 7f5d0d2bb700 1 -- start start 2023-11-30T21:52:05.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.205+0000 7f5d0d2bb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080f5410 0x7f5d08106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:05.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.205+0000 7f5d0d2bb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d08106c50 con 0x7f5d080f5410 2023-11-30T21:52:05.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.205+0000 7f5d06ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080f5410 0x7f5d08106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:05.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.205+0000 7f5d06ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080f5410 0x7f5d08106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:58850/0 (socket says 172.21.15.49:58850) 2023-11-30T21:52:05.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.205+0000 7f5d06ffd700 1 -- 172.21.15.49:0/1494875703 learned_addr learned my addr 172.21.15.49:0/1494875703 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:05.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.206+0000 7f5d06ffd700 1 -- 172.21.15.49:0/1494875703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cfc005040 con 0x7f5d080f5410 2023-11-30T21:52:05.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.206+0000 7f5d06ffd700 1 --2- 172.21.15.49:0/1494875703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080f5410 0x7f5d08106760 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f5cfc009f80 tx=0x7f5cfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:05.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.206+0000 7f5cef7fe700 1 -- 172.21.15.49:0/1494875703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cfc009320 con 0x7f5d080f5410 2023-11-30T21:52:05.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.206+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/1494875703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d08106e50 con 0x7f5d080f5410 2023-11-30T21:52:05.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.206+0000 7f5cef7fe700 1 -- 172.21.15.49:0/1494875703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5cfc013ec0 con 0x7f5d080f5410 2023-11-30T21:52:05.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.206+0000 7f5cef7fe700 1 -- 172.21.15.49:0/1494875703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cfc01ece0 con 0x7f5d080f5410 2023-11-30T21:52:05.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.206+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/1494875703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d081072e0 con 0x7f5d080f5410 2023-11-30T21:52:05.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.207+0000 7f5cef7fe700 1 -- 172.21.15.49:0/1494875703 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5cfc014420 con 0x7f5d080f5410 2023-11-30T21:52:05.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.207+0000 7f5cef7fe700 1 --2- 172.21.15.49:0/1494875703 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5cf002ff60 0x7f5cf0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:05.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.207+0000 7f5cef7fe700 1 -- 172.21.15.49:0/1494875703 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5cfc040d20 con 0x7f5d080f5410 2023-11-30T21:52:05.211 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.207+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/1494875703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d08045ad0 con 0x7f5d080f5410 2023-11-30T21:52:05.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.207+0000 7f5d067fc700 1 --2- 172.21.15.49:0/1494875703 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5cf002ff60 0x7f5cf0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:05.212 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.208+0000 7f5d067fc700 1 --2- 172.21.15.49:0/1494875703 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5cf002ff60 0x7f5cf0032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f5cf80096a0 tx=0x7f5cf8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:05.213 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.212+0000 7f5cef7fe700 1 -- 172.21.15.49:0/1494875703 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5cfc01c8f0 con 0x7f5d080f5410 2023-11-30T21:52:05.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.430+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/1494875703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d08108000 con 0x7f5d080f5410 2023-11-30T21:52:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.431+0000 7f5cef7fe700 1 -- 172.21.15.49:0/1494875703 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5cfc01c320 con 0x7f5d080f5410 2023-11-30T21:52:05.433 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:05.433 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:05.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.432+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/1494875703 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5cf002ff60 msgr2=0x7f5cf0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:05.434 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.432+0000 7f5d0d2bb700 1 --2- 172.21.15.49:0/1494875703 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5cf002ff60 0x7f5cf0032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f5cf80096a0 tx=0x7f5cf8006b40).stop 2023-11-30T21:52:05.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.433+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/1494875703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080f5410 msgr2=0x7f5d08106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:05.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.433+0000 7f5d0d2bb700 1 --2- 172.21.15.49:0/1494875703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080f5410 0x7f5d08106760 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f5cfc009f80 tx=0x7f5cfc00b040).stop 2023-11-30T21:52:05.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.433+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/1494875703 shutdown_connections 2023-11-30T21:52:05.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.433+0000 7f5d0d2bb700 1 --2- 172.21.15.49:0/1494875703 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5cf002ff60 0x7f5cf0032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:05.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.433+0000 7f5d0d2bb700 1 --2- 172.21.15.49:0/1494875703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d080f5410 0x7f5d08106760 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:05.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.433+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/1494875703 >> 172.21.15.49:0/1494875703 conn(0x7f5d080f0b70 msgr2=0x7f5d0805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:05.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.433+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/1494875703 shutdown_connections 2023-11-30T21:52:05.435 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:05.433+0000 7f5d0d2bb700 1 -- 172.21.15.49:0/1494875703 wait complete. 2023-11-30T21:52:05.435 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:05.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:03.534950+0000 mgr.smithi027.brrjcl (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:06.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:06 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:05.432656+0000 mon.smithi027 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.49:0/1494875703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:06.858 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:06.859 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:07.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:07 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:05.535342+0000 mgr.smithi027.brrjcl (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:08.137 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.135+0000 7fc8022a3700 1 -- 172.21.15.49:0/3069109720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7fc0f78e0 msgr2=0x7fc7fc0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:08.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.135+0000 7fc8022a3700 1 --2- 172.21.15.49:0/3069109720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7fc0f78e0 0x7fc7fc0f7d00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fc7f8009fb0 tx=0x7fc7f8009b30).stop 2023-11-30T21:52:08.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.135+0000 7fc8022a3700 1 -- 172.21.15.49:0/3069109720 shutdown_connections 2023-11-30T21:52:08.138 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.135+0000 7fc8022a3700 1 --2- 172.21.15.49:0/3069109720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7fc0f78e0 0x7fc7fc0f7d00 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:08.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.135+0000 7fc8022a3700 1 -- 172.21.15.49:0/3069109720 >> 172.21.15.49:0/3069109720 conn(0x7fc7fc0f0120 msgr2=0x7fc7fc0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:08.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.135+0000 7fc8022a3700 1 -- 172.21.15.49:0/3069109720 shutdown_connections 2023-11-30T21:52:08.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.135+0000 7fc8022a3700 1 -- 172.21.15.49:0/3069109720 wait complete. 2023-11-30T21:52:08.139 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.135+0000 7fc8022a3700 1 Processor -- start 2023-11-30T21:52:08.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.135+0000 7fc8022a3700 1 -- start start 2023-11-30T21:52:08.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.136+0000 7fc8022a3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7fc0f78e0 0x7fc7fc1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:08.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.136+0000 7fc8022a3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7fc1069d0 con 0x7fc7fc0f78e0 2023-11-30T21:52:08.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.136+0000 7fc8012a1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7fc0f78e0 0x7fc7fc1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:08.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.136+0000 7fc8012a1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7fc0f78e0 0x7fc7fc1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:58870/0 (socket says 172.21.15.49:58870) 2023-11-30T21:52:08.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.136+0000 7fc8012a1700 1 -- 172.21.15.49:0/3362298503 learned_addr learned my addr 172.21.15.49:0/3362298503 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:08.142 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.136+0000 7fc8012a1700 1 -- 172.21.15.49:0/3362298503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f8005040 con 0x7fc7fc0f78e0 2023-11-30T21:52:08.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.136+0000 7fc8012a1700 1 --2- 172.21.15.49:0/3362298503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7fc0f78e0 0x7fc7fc1064e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fc7f8013510 tx=0x7fc7f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:08.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.136+0000 7fc7f1ffb700 1 -- 172.21.15.49:0/3362298503 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7f8009320 con 0x7fc7fc0f78e0 2023-11-30T21:52:08.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.136+0000 7fc8022a3700 1 -- 172.21.15.49:0/3362298503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7fc106bd0 con 0x7fc7fc0f78e0 2023-11-30T21:52:08.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.136+0000 7fc7f1ffb700 1 -- 172.21.15.49:0/3362298503 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7f8013f00 con 0x7fc7fc0f78e0 2023-11-30T21:52:08.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.137+0000 7fc7f1ffb700 1 -- 172.21.15.49:0/3362298503 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7f8014dc0 con 0x7fc7fc0f78e0 2023-11-30T21:52:08.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.137+0000 7fc8022a3700 1 -- 172.21.15.49:0/3362298503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7fc107060 con 0x7fc7fc0f78e0 2023-11-30T21:52:08.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.138+0000 7fc8022a3700 1 -- 172.21.15.49:0/3362298503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7fc045ad0 con 0x7fc7fc0f78e0 2023-11-30T21:52:08.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.138+0000 7fc7f1ffb700 1 -- 172.21.15.49:0/3362298503 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7f80095f0 con 0x7fc7fc0f78e0 2023-11-30T21:52:08.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.138+0000 7fc7f1ffb700 1 --2- 172.21.15.49:0/3362298503 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc7e802ffb0 0x7fc7e8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:08.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.138+0000 7fc7f1ffb700 1 -- 172.21.15.49:0/3362298503 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7f80410b0 con 0x7fc7fc0f78e0 2023-11-30T21:52:08.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.138+0000 7fc800aa0700 1 --2- 172.21.15.49:0/3362298503 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc7e802ffb0 0x7fc7e8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:08.146 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.139+0000 7fc800aa0700 1 --2- 172.21.15.49:0/3362298503 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc7e802ffb0 0x7fc7e8032470 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fc7ec0096a0 tx=0x7fc7ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:08.146 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.143+0000 7fc7f1ffb700 1 -- 172.21.15.49:0/3362298503 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7f806d2b0 con 0x7fc7fc0f78e0 2023-11-30T21:52:08.362 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.361+0000 7fc8022a3700 1 -- 172.21.15.49:0/3362298503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7fc107f10 con 0x7fc7fc0f78e0 2023-11-30T21:52:08.363 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.361+0000 7fc7f1ffb700 1 -- 172.21.15.49:0/3362298503 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc7f8014420 con 0x7fc7fc0f78e0 2023-11-30T21:52:08.363 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:08.363 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:08.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.363+0000 7fc8022a3700 1 -- 172.21.15.49:0/3362298503 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc7e802ffb0 msgr2=0x7fc7e8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:08.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.363+0000 7fc8022a3700 1 --2- 172.21.15.49:0/3362298503 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc7e802ffb0 0x7fc7e8032470 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fc7ec0096a0 tx=0x7fc7ec006b40).stop 2023-11-30T21:52:08.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.363+0000 7fc8022a3700 1 -- 172.21.15.49:0/3362298503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7fc0f78e0 msgr2=0x7fc7fc1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:08.366 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.363+0000 7fc8022a3700 1 --2- 172.21.15.49:0/3362298503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7fc0f78e0 0x7fc7fc1064e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fc7f8013510 tx=0x7fc7f800b040).stop 2023-11-30T21:52:08.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.363+0000 7fc8022a3700 1 -- 172.21.15.49:0/3362298503 shutdown_connections 2023-11-30T21:52:08.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.363+0000 7fc8022a3700 1 --2- 172.21.15.49:0/3362298503 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fc7e802ffb0 0x7fc7e8032470 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:08.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.363+0000 7fc8022a3700 1 --2- 172.21.15.49:0/3362298503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7fc0f78e0 0x7fc7fc1064e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:08.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.363+0000 7fc8022a3700 1 -- 172.21.15.49:0/3362298503 >> 172.21.15.49:0/3362298503 conn(0x7fc7fc0f0120 msgr2=0x7fc7fc0f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:08.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.363+0000 7fc8022a3700 1 -- 172.21.15.49:0/3362298503 shutdown_connections 2023-11-30T21:52:08.367 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:08.363+0000 7fc8022a3700 1 -- 172.21.15.49:0/3362298503 wait complete. 2023-11-30T21:52:08.367 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:08.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:08 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:07.535740+0000 mgr.smithi027.brrjcl (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:09.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:09 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:08.362859+0000 mon.smithi027 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.49:0/3362298503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:09.799 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:09.799 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:10.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:10 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:09.536076+0000 mgr.smithi027.brrjcl (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:11.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.094+0000 7fdc47af1700 1 -- 172.21.15.49:0/4221223117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40060280 msgr2=0x7fdc400f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:11.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.094+0000 7fdc47af1700 1 --2- 172.21.15.49:0/4221223117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40060280 0x7fdc400f90f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fdc3c009fb0 tx=0x7fdc3c009b30).stop 2023-11-30T21:52:11.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.094+0000 7fdc47af1700 1 -- 172.21.15.49:0/4221223117 shutdown_connections 2023-11-30T21:52:11.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.095+0000 7fdc47af1700 1 --2- 172.21.15.49:0/4221223117 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40060280 0x7fdc400f90f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:11.098 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.095+0000 7fdc47af1700 1 -- 172.21.15.49:0/4221223117 >> 172.21.15.49:0/4221223117 conn(0x7fdc400f0b70 msgr2=0x7fdc400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.095+0000 7fdc47af1700 1 -- 172.21.15.49:0/4221223117 shutdown_connections 2023-11-30T21:52:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.095+0000 7fdc47af1700 1 -- 172.21.15.49:0/4221223117 wait complete. 2023-11-30T21:52:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.095+0000 7fdc47af1700 1 Processor -- start 2023-11-30T21:52:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.095+0000 7fdc47af1700 1 -- start start 2023-11-30T21:52:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.095+0000 7fdc47af1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40060280 0x7fdc401066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.095+0000 7fdc47af1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc40106bd0 con 0x7fdc40060280 2023-11-30T21:52:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.096+0000 7fdc4588d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40060280 0x7fdc401066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.096+0000 7fdc4588d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40060280 0x7fdc401066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:58508/0 (socket says 172.21.15.49:58508) 2023-11-30T21:52:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.096+0000 7fdc4588d700 1 -- 172.21.15.49:0/667102000 learned_addr learned my addr 172.21.15.49:0/667102000 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.096+0000 7fdc4588d700 1 -- 172.21.15.49:0/667102000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c005040 con 0x7fdc40060280 2023-11-30T21:52:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.096+0000 7fdc4588d700 1 --2- 172.21.15.49:0/667102000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40060280 0x7fdc401066e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fdc3c013510 tx=0x7fdc3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.096+0000 7fdc327fc700 1 -- 172.21.15.49:0/667102000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc3c009320 con 0x7fdc40060280 2023-11-30T21:52:11.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.096+0000 7fdc47af1700 1 -- 172.21.15.49:0/667102000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc40106dd0 con 0x7fdc40060280 2023-11-30T21:52:11.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.096+0000 7fdc327fc700 1 -- 172.21.15.49:0/667102000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc3c013f00 con 0x7fdc40060280 2023-11-30T21:52:11.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.096+0000 7fdc327fc700 1 -- 172.21.15.49:0/667102000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc3c014dc0 con 0x7fdc40060280 2023-11-30T21:52:11.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.096+0000 7fdc47af1700 1 -- 172.21.15.49:0/667102000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc40107260 con 0x7fdc40060280 2023-11-30T21:52:11.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.097+0000 7fdc327fc700 1 -- 172.21.15.49:0/667102000 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc3c0095f0 con 0x7fdc40060280 2023-11-30T21:52:11.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.097+0000 7fdc47af1700 1 -- 172.21.15.49:0/667102000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc40045ad0 con 0x7fdc40060280 2023-11-30T21:52:11.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.097+0000 7fdc327fc700 1 --2- 172.21.15.49:0/667102000 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdc2c02ff60 0x7fdc2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:11.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.097+0000 7fdc327fc700 1 -- 172.21.15.49:0/667102000 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc3c041040 con 0x7fdc40060280 2023-11-30T21:52:11.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.098+0000 7fdc4508c700 1 --2- 172.21.15.49:0/667102000 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdc2c02ff60 0x7fdc2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:11.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.098+0000 7fdc4508c700 1 --2- 172.21.15.49:0/667102000 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdc2c02ff60 0x7fdc2c032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fdc340096a0 tx=0x7fdc34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:11.103 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.102+0000 7fdc327fc700 1 -- 172.21.15.49:0/667102000 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc3c014420 con 0x7fdc40060280 2023-11-30T21:52:11.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.319+0000 7fdc47af1700 1 -- 172.21.15.49:0/667102000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc40108110 con 0x7fdc40060280 2023-11-30T21:52:11.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.319+0000 7fdc327fc700 1 -- 172.21.15.49:0/667102000 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc3c010230 con 0x7fdc40060280 2023-11-30T21:52:11.323 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:11.323 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:11.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.321+0000 7fdc47af1700 1 -- 172.21.15.49:0/667102000 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdc2c02ff60 msgr2=0x7fdc2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:11.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.321+0000 7fdc47af1700 1 --2- 172.21.15.49:0/667102000 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdc2c02ff60 0x7fdc2c032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fdc340096a0 tx=0x7fdc34006b40).stop 2023-11-30T21:52:11.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.321+0000 7fdc47af1700 1 -- 172.21.15.49:0/667102000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40060280 msgr2=0x7fdc401066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:11.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.321+0000 7fdc47af1700 1 --2- 172.21.15.49:0/667102000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40060280 0x7fdc401066e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fdc3c013510 tx=0x7fdc3c00b040).stop 2023-11-30T21:52:11.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.321+0000 7fdc47af1700 1 -- 172.21.15.49:0/667102000 shutdown_connections 2023-11-30T21:52:11.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.321+0000 7fdc47af1700 1 --2- 172.21.15.49:0/667102000 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fdc2c02ff60 0x7fdc2c032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:11.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.321+0000 7fdc47af1700 1 --2- 172.21.15.49:0/667102000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc40060280 0x7fdc401066e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:11.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.321+0000 7fdc47af1700 1 -- 172.21.15.49:0/667102000 >> 172.21.15.49:0/667102000 conn(0x7fdc400f0b70 msgr2=0x7fdc400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:11.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.321+0000 7fdc47af1700 1 -- 172.21.15.49:0/667102000 shutdown_connections 2023-11-30T21:52:11.324 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:11.321+0000 7fdc47af1700 1 -- 172.21.15.49:0/667102000 wait complete. 2023-11-30T21:52:11.324 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:11.761 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:11.320995+0000 mon.smithi027 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.49:0/667102000' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:12.714 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:12.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:12.760 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:12 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:11.536325+0000 mgr.smithi027.brrjcl (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:13.914 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.912+0000 7f8a6924a700 1 -- 172.21.15.49:0/2576044917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a640f5410 msgr2=0x7f8a640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:13.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.912+0000 7f8a6924a700 1 --2- 172.21.15.49:0/2576044917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a640f5410 0x7f8a640f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f8a54009fb0 tx=0x7f8a54009b30).stop 2023-11-30T21:52:13.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.912+0000 7f8a6924a700 1 -- 172.21.15.49:0/2576044917 shutdown_connections 2023-11-30T21:52:13.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.912+0000 7f8a6924a700 1 --2- 172.21.15.49:0/2576044917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a640f5410 0x7f8a640f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:13.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.912+0000 7f8a6924a700 1 -- 172.21.15.49:0/2576044917 >> 172.21.15.49:0/2576044917 conn(0x7f8a640f0b70 msgr2=0x7f8a640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:13.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.912+0000 7f8a6924a700 1 -- 172.21.15.49:0/2576044917 shutdown_connections 2023-11-30T21:52:13.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.912+0000 7f8a6924a700 1 -- 172.21.15.49:0/2576044917 wait complete. 2023-11-30T21:52:13.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.913+0000 7f8a6924a700 1 Processor -- start 2023-11-30T21:52:13.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.913+0000 7f8a6924a700 1 -- start start 2023-11-30T21:52:13.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.913+0000 7f8a6924a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a640f5410 0x7f8a640fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:13.915 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.913+0000 7f8a6924a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a640fb820 con 0x7f8a640f5410 2023-11-30T21:52:13.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.913+0000 7f8a62d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a640f5410 0x7f8a640fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:13.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.913+0000 7f8a62d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a640f5410 0x7f8a640fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:58528/0 (socket says 172.21.15.49:58528) 2023-11-30T21:52:13.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.913+0000 7f8a62d9d700 1 -- 172.21.15.49:0/2024238286 learned_addr learned my addr 172.21.15.49:0/2024238286 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:13.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.913+0000 7f8a62d9d700 1 -- 172.21.15.49:0/2024238286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a54005040 con 0x7f8a640f5410 2023-11-30T21:52:13.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.914+0000 7f8a62d9d700 1 --2- 172.21.15.49:0/2024238286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a640f5410 0x7f8a640fd3a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f8a54013510 tx=0x7f8a5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:13.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.914+0000 7f8a4b7fe700 1 -- 172.21.15.49:0/2024238286 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a54009320 con 0x7f8a640f5410 2023-11-30T21:52:13.917 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.914+0000 7f8a4b7fe700 1 -- 172.21.15.49:0/2024238286 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a54013f00 con 0x7f8a640f5410 2023-11-30T21:52:13.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.914+0000 7f8a6924a700 1 -- 172.21.15.49:0/2024238286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a640fb9c0 con 0x7f8a640f5410 2023-11-30T21:52:13.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.914+0000 7f8a4b7fe700 1 -- 172.21.15.49:0/2024238286 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a54014d90 con 0x7f8a640f5410 2023-11-30T21:52:13.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.914+0000 7f8a6924a700 1 -- 172.21.15.49:0/2024238286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a640fbe50 con 0x7f8a640f5410 2023-11-30T21:52:13.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.915+0000 7f8a6924a700 1 -- 172.21.15.49:0/2024238286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a64059100 con 0x7f8a640f5410 2023-11-30T21:52:13.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.915+0000 7f8a4b7fe700 1 -- 172.21.15.49:0/2024238286 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a540095f0 con 0x7f8a640f5410 2023-11-30T21:52:13.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.915+0000 7f8a4b7fe700 1 --2- 172.21.15.49:0/2024238286 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8a4c02ff60 0x7f8a4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:13.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.915+0000 7f8a4b7fe700 1 -- 172.21.15.49:0/2024238286 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a54040fe0 con 0x7f8a640f5410 2023-11-30T21:52:13.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.915+0000 7f8a6259c700 1 --2- 172.21.15.49:0/2024238286 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8a4c02ff60 0x7f8a4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:13.918 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.916+0000 7f8a6259c700 1 --2- 172.21.15.49:0/2024238286 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8a4c02ff60 0x7f8a4c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f8a580096a0 tx=0x7f8a58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:13.921 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:13.920+0000 7f8a4b7fe700 1 -- 172.21.15.49:0/2024238286 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a5401d7f0 con 0x7f8a640f5410 2023-11-30T21:52:14.140 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:14.139+0000 7f8a6924a700 1 -- 172.21.15.49:0/2024238286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a640fae60 con 0x7f8a640f5410 2023-11-30T21:52:14.141 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:14.140+0000 7f8a4b7fe700 1 -- 172.21.15.49:0/2024238286 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a54010230 con 0x7f8a640f5410 2023-11-30T21:52:14.142 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:14.142 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:14.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:14.141+0000 7f8a6924a700 1 -- 172.21.15.49:0/2024238286 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8a4c02ff60 msgr2=0x7f8a4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:14.143 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:14.141+0000 7f8a6924a700 1 --2- 172.21.15.49:0/2024238286 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8a4c02ff60 0x7f8a4c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f8a580096a0 tx=0x7f8a58006b40).stop 2023-11-30T21:52:14.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:14.141+0000 7f8a6924a700 1 -- 172.21.15.49:0/2024238286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a640f5410 msgr2=0x7f8a640fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:14.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:14.141+0000 7f8a6924a700 1 --2- 172.21.15.49:0/2024238286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a640f5410 0x7f8a640fd3a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f8a54013510 tx=0x7f8a5400b040).stop 2023-11-30T21:52:14.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:14.141+0000 7f8a6924a700 1 -- 172.21.15.49:0/2024238286 shutdown_connections 2023-11-30T21:52:14.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:14.141+0000 7f8a6924a700 1 --2- 172.21.15.49:0/2024238286 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8a4c02ff60 0x7f8a4c032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:14.144 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:14.141+0000 7f8a6924a700 1 --2- 172.21.15.49:0/2024238286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a640f5410 0x7f8a640fd3a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:14.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:14.141+0000 7f8a6924a700 1 -- 172.21.15.49:0/2024238286 >> 172.21.15.49:0/2024238286 conn(0x7f8a640f0b70 msgr2=0x7f8a640f1770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:14.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:14.141+0000 7f8a6924a700 1 -- 172.21.15.49:0/2024238286 shutdown_connections 2023-11-30T21:52:14.145 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:14.141+0000 7f8a6924a700 1 -- 172.21.15.49:0/2024238286 wait complete. 2023-11-30T21:52:14.145 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:15.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:15 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:13.536681+0000 mgr.smithi027.brrjcl (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:15.512 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:15 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:14.141145+0000 mon.smithi027 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.49:0/2024238286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:15.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:15.531 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:16.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.682+0000 7f29fc497700 1 -- 172.21.15.49:0/689742651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f40f5410 msgr2=0x7f29f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:16.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.682+0000 7f29fc497700 1 --2- 172.21.15.49:0/689742651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f40f5410 0x7f29f40f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f29f0009fb0 tx=0x7f29f0009b30).stop 2023-11-30T21:52:16.684 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.682+0000 7f29fc497700 1 -- 172.21.15.49:0/689742651 shutdown_connections 2023-11-30T21:52:16.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.682+0000 7f29fc497700 1 --2- 172.21.15.49:0/689742651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f40f5410 0x7f29f40f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:16.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.682+0000 7f29fc497700 1 -- 172.21.15.49:0/689742651 >> 172.21.15.49:0/689742651 conn(0x7f29f40f0b70 msgr2=0x7f29f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:16.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.682+0000 7f29fc497700 1 -- 172.21.15.49:0/689742651 shutdown_connections 2023-11-30T21:52:16.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.682+0000 7f29fc497700 1 -- 172.21.15.49:0/689742651 wait complete. 2023-11-30T21:52:16.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.682+0000 7f29fc497700 1 Processor -- start 2023-11-30T21:52:16.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.682+0000 7f29fc497700 1 -- start start 2023-11-30T21:52:16.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.682+0000 7f29fc497700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f40f5410 0x7f29f4108930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:16.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.682+0000 7f29fc497700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f4108e20 con 0x7f29f40f5410 2023-11-30T21:52:16.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.683+0000 7f29fa233700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f40f5410 0x7f29f4108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:16.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.683+0000 7f29fa233700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f40f5410 0x7f29f4108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:58538/0 (socket says 172.21.15.49:58538) 2023-11-30T21:52:16.685 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.683+0000 7f29fa233700 1 -- 172.21.15.49:0/2759314702 learned_addr learned my addr 172.21.15.49:0/2759314702 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:16.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.683+0000 7f29fa233700 1 -- 172.21.15.49:0/2759314702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f0005040 con 0x7f29f40f5410 2023-11-30T21:52:16.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.683+0000 7f29fa233700 1 --2- 172.21.15.49:0/2759314702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f40f5410 0x7f29f4108930 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f29f0013510 tx=0x7f29f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:16.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.683+0000 7f29e6ffd700 1 -- 172.21.15.49:0/2759314702 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29f0009320 con 0x7f29f40f5410 2023-11-30T21:52:16.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.683+0000 7f29fc497700 1 -- 172.21.15.49:0/2759314702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f4109020 con 0x7f29f40f5410 2023-11-30T21:52:16.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.683+0000 7f29e6ffd700 1 -- 172.21.15.49:0/2759314702 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29f0013f00 con 0x7f29f40f5410 2023-11-30T21:52:16.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.683+0000 7f29e6ffd700 1 -- 172.21.15.49:0/2759314702 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29f0014d90 con 0x7f29f40f5410 2023-11-30T21:52:16.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.683+0000 7f29fc497700 1 -- 172.21.15.49:0/2759314702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f41094b0 con 0x7f29f40f5410 2023-11-30T21:52:16.687 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.684+0000 7f29e6ffd700 1 -- 172.21.15.49:0/2759314702 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29f0014420 con 0x7f29f40f5410 2023-11-30T21:52:16.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.684+0000 7f29fc497700 1 -- 172.21.15.49:0/2759314702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f4045ad0 con 0x7f29f40f5410 2023-11-30T21:52:16.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.684+0000 7f29e6ffd700 1 --2- 172.21.15.49:0/2759314702 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f29e002ffb0 0x7f29e0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:16.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.684+0000 7f29e6ffd700 1 -- 172.21.15.49:0/2759314702 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29f0041090 con 0x7f29f40f5410 2023-11-30T21:52:16.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.684+0000 7f29f9a32700 1 --2- 172.21.15.49:0/2759314702 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f29e002ffb0 0x7f29e0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:16.688 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.685+0000 7f29f9a32700 1 --2- 172.21.15.49:0/2759314702 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f29e002ffb0 0x7f29e0032470 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f29e80096a0 tx=0x7f29e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:16.690 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.689+0000 7f29e6ffd700 1 -- 172.21.15.49:0/2759314702 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29f006d2b0 con 0x7f29f40f5410 2023-11-30T21:52:16.909 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.908+0000 7f29fc497700 1 -- 172.21.15.49:0/2759314702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29f40fe190 con 0x7f29f40f5410 2023-11-30T21:52:16.910 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.908+0000 7f29e6ffd700 1 -- 172.21.15.49:0/2759314702 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f29f00095f0 con 0x7f29f40f5410 2023-11-30T21:52:16.910 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:16.910 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:16.911 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.910+0000 7f29fc497700 1 -- 172.21.15.49:0/2759314702 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f29e002ffb0 msgr2=0x7f29e0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:16.911 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.910+0000 7f29fc497700 1 --2- 172.21.15.49:0/2759314702 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f29e002ffb0 0x7f29e0032470 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f29e80096a0 tx=0x7f29e8006b40).stop 2023-11-30T21:52:16.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.910+0000 7f29fc497700 1 -- 172.21.15.49:0/2759314702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f40f5410 msgr2=0x7f29f4108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:16.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.910+0000 7f29fc497700 1 --2- 172.21.15.49:0/2759314702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f40f5410 0x7f29f4108930 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f29f0013510 tx=0x7f29f000b040).stop 2023-11-30T21:52:16.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.910+0000 7f29fc497700 1 -- 172.21.15.49:0/2759314702 shutdown_connections 2023-11-30T21:52:16.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.910+0000 7f29fc497700 1 --2- 172.21.15.49:0/2759314702 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f29e002ffb0 0x7f29e0032470 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:16.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.910+0000 7f29fc497700 1 --2- 172.21.15.49:0/2759314702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29f40f5410 0x7f29f4108930 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:16.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.910+0000 7f29fc497700 1 -- 172.21.15.49:0/2759314702 >> 172.21.15.49:0/2759314702 conn(0x7f29f40f0b70 msgr2=0x7f29f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:16.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.910+0000 7f29fc497700 1 -- 172.21.15.49:0/2759314702 shutdown_connections 2023-11-30T21:52:16.912 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:16.910+0000 7f29fc497700 1 -- 172.21.15.49:0/2759314702 wait complete. 2023-11-30T21:52:16.912 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:17.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:15.537092+0000 mgr.smithi027.brrjcl (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:17.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:16.909784+0000 mon.smithi027 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.49:0/2759314702' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:18.300 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:18.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:19.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:17.537426+0000 mgr.smithi027.brrjcl (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:19.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.520+0000 7fae2db49700 1 -- 172.21.15.49:0/3137612313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae280f51a0 msgr2=0x7fae280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:19.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.520+0000 7fae2db49700 1 --2- 172.21.15.49:0/3137612313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae280f51a0 0x7fae280f55c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fae1c009fb0 tx=0x7fae1c009b30).stop 2023-11-30T21:52:19.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.520+0000 7fae2db49700 1 -- 172.21.15.49:0/3137612313 shutdown_connections 2023-11-30T21:52:19.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.520+0000 7fae2db49700 1 --2- 172.21.15.49:0/3137612313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae280f51a0 0x7fae280f55c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:19.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.520+0000 7fae2db49700 1 -- 172.21.15.49:0/3137612313 >> 172.21.15.49:0/3137612313 conn(0x7fae280f09a0 msgr2=0x7fae280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:19.521 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.520+0000 7fae2db49700 1 -- 172.21.15.49:0/3137612313 shutdown_connections 2023-11-30T21:52:19.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.520+0000 7fae2db49700 1 -- 172.21.15.49:0/3137612313 wait complete. 2023-11-30T21:52:19.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.520+0000 7fae2db49700 1 Processor -- start 2023-11-30T21:52:19.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.520+0000 7fae2db49700 1 -- start start 2023-11-30T21:52:19.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.521+0000 7fae2db49700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae280f51a0 0x7fae281091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:19.522 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.521+0000 7fae2db49700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae28060240 con 0x7fae280f51a0 2023-11-30T21:52:19.523 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.521+0000 7fae2cb47700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae280f51a0 0x7fae281091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:19.523 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.521+0000 7fae2cb47700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae280f51a0 0x7fae281091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:37712/0 (socket says 172.21.15.49:37712) 2023-11-30T21:52:19.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.521+0000 7fae2cb47700 1 -- 172.21.15.49:0/2954529270 learned_addr learned my addr 172.21.15.49:0/2954529270 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:19.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.521+0000 7fae2cb47700 1 -- 172.21.15.49:0/2954529270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c005040 con 0x7fae280f51a0 2023-11-30T21:52:19.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.521+0000 7fae2cb47700 1 --2- 172.21.15.49:0/2954529270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae280f51a0 0x7fae281091d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fae1c013510 tx=0x7fae1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:19.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.522+0000 7fae257fa700 1 -- 172.21.15.49:0/2954529270 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae1c009320 con 0x7fae280f51a0 2023-11-30T21:52:19.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.522+0000 7fae257fa700 1 -- 172.21.15.49:0/2954529270 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae1c013f00 con 0x7fae280f51a0 2023-11-30T21:52:19.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.522+0000 7fae2db49700 1 -- 172.21.15.49:0/2954529270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae28060440 con 0x7fae280f51a0 2023-11-30T21:52:19.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.522+0000 7fae257fa700 1 -- 172.21.15.49:0/2954529270 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae1c014dc0 con 0x7fae280f51a0 2023-11-30T21:52:19.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.522+0000 7fae2db49700 1 -- 172.21.15.49:0/2954529270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae28109880 con 0x7fae280f51a0 2023-11-30T21:52:19.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.523+0000 7fae257fa700 1 -- 172.21.15.49:0/2954529270 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae1c0095f0 con 0x7fae280f51a0 2023-11-30T21:52:19.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.523+0000 7fae2db49700 1 -- 172.21.15.49:0/2954529270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae28045ad0 con 0x7fae280f51a0 2023-11-30T21:52:19.524 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.523+0000 7fae257fa700 1 --2- 172.21.15.49:0/2954529270 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fae1802ffb0 0x7fae18032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:19.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.523+0000 7fae257fa700 1 -- 172.21.15.49:0/2954529270 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae1c040fb0 con 0x7fae280f51a0 2023-11-30T21:52:19.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.523+0000 7fae27fff700 1 --2- 172.21.15.49:0/2954529270 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fae1802ffb0 0x7fae18032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:19.525 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.523+0000 7fae27fff700 1 --2- 172.21.15.49:0/2954529270 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fae1802ffb0 0x7fae18032470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fae140096a0 tx=0x7fae14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:19.528 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.528+0000 7fae257fa700 1 -- 172.21.15.49:0/2954529270 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae1c06d2b0 con 0x7fae280f51a0 2023-11-30T21:52:19.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.748+0000 7fae2db49700 1 -- 172.21.15.49:0/2954529270 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae2810a0a0 con 0x7fae280f51a0 2023-11-30T21:52:19.749 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.749+0000 7fae257fa700 1 -- 172.21.15.49:0/2954529270 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fae1c014420 con 0x7fae280f51a0 2023-11-30T21:52:19.750 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:19.750 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:19.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.750+0000 7fae2db49700 1 -- 172.21.15.49:0/2954529270 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fae1802ffb0 msgr2=0x7fae18032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:19.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.750+0000 7fae2db49700 1 --2- 172.21.15.49:0/2954529270 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fae1802ffb0 0x7fae18032470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fae140096a0 tx=0x7fae14006b40).stop 2023-11-30T21:52:19.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.750+0000 7fae2db49700 1 -- 172.21.15.49:0/2954529270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae280f51a0 msgr2=0x7fae281091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:19.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.750+0000 7fae2db49700 1 --2- 172.21.15.49:0/2954529270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae280f51a0 0x7fae281091d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fae1c013510 tx=0x7fae1c00b040).stop 2023-11-30T21:52:19.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.750+0000 7fae2db49700 1 -- 172.21.15.49:0/2954529270 shutdown_connections 2023-11-30T21:52:19.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.750+0000 7fae2db49700 1 --2- 172.21.15.49:0/2954529270 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fae1802ffb0 0x7fae18032470 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:19.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.750+0000 7fae2db49700 1 --2- 172.21.15.49:0/2954529270 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae280f51a0 0x7fae281091d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:19.751 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.750+0000 7fae2db49700 1 -- 172.21.15.49:0/2954529270 >> 172.21.15.49:0/2954529270 conn(0x7fae280f09a0 msgr2=0x7fae280f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:19.752 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.750+0000 7fae2db49700 1 -- 172.21.15.49:0/2954529270 shutdown_connections 2023-11-30T21:52:19.752 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:19.750+0000 7fae2db49700 1 -- 172.21.15.49:0/2954529270 wait complete. 2023-11-30T21:52:19.752 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:20.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:20 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:19.749296+0000 mon.smithi027 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.49:0/2954529270' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:21.165 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:21.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:21.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:21 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:19.537763+0000 mgr.smithi027.brrjcl (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:22.400 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.400+0000 7fa469c61700 1 -- 172.21.15.49:0/2035337384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa464060280 msgr2=0x7fa4640f8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:22.400 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.400+0000 7fa469c61700 1 --2- 172.21.15.49:0/2035337384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa464060280 0x7fa4640f8ea0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fa454009fb0 tx=0x7fa454009b30).stop 2023-11-30T21:52:22.401 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.400+0000 7fa469c61700 1 -- 172.21.15.49:0/2035337384 shutdown_connections 2023-11-30T21:52:22.401 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.400+0000 7fa469c61700 1 --2- 172.21.15.49:0/2035337384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa464060280 0x7fa4640f8ea0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:22.401 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.400+0000 7fa469c61700 1 -- 172.21.15.49:0/2035337384 >> 172.21.15.49:0/2035337384 conn(0x7fa4640f0920 msgr2=0x7fa4640f2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:22.401 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.400+0000 7fa469c61700 1 -- 172.21.15.49:0/2035337384 shutdown_connections 2023-11-30T21:52:22.401 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.400+0000 7fa469c61700 1 -- 172.21.15.49:0/2035337384 wait complete. 2023-11-30T21:52:22.402 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.400+0000 7fa469c61700 1 Processor -- start 2023-11-30T21:52:22.402 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.400+0000 7fa469c61700 1 -- start start 2023-11-30T21:52:22.403 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.401+0000 7fa469c61700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa464060280 0x7fa464106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:22.403 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.401+0000 7fa469c61700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4641069f0 con 0x7fa464060280 2023-11-30T21:52:22.403 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.401+0000 7fa4637fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa464060280 0x7fa464106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:22.403 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.401+0000 7fa4637fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa464060280 0x7fa464106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:37720/0 (socket says 172.21.15.49:37720) 2023-11-30T21:52:22.403 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.401+0000 7fa4637fe700 1 -- 172.21.15.49:0/891102258 learned_addr learned my addr 172.21.15.49:0/891102258 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:22.403 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.401+0000 7fa4637fe700 1 -- 172.21.15.49:0/891102258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa454005040 con 0x7fa464060280 2023-11-30T21:52:22.403 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.401+0000 7fa4637fe700 1 --2- 172.21.15.49:0/891102258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa464060280 0x7fa464106500 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fa454013510 tx=0x7fa45400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:22.403 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.401+0000 7fa44bfff700 1 -- 172.21.15.49:0/891102258 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa454009320 con 0x7fa464060280 2023-11-30T21:52:22.403 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.401+0000 7fa469c61700 1 -- 172.21.15.49:0/891102258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa464106bf0 con 0x7fa464060280 2023-11-30T21:52:22.403 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.401+0000 7fa44bfff700 1 -- 172.21.15.49:0/891102258 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa454013f00 con 0x7fa464060280 2023-11-30T21:52:22.403 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.401+0000 7fa44bfff700 1 -- 172.21.15.49:0/891102258 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa454014dc0 con 0x7fa464060280 2023-11-30T21:52:22.403 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.402+0000 7fa469c61700 1 -- 172.21.15.49:0/891102258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa464107080 con 0x7fa464060280 2023-11-30T21:52:22.404 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.402+0000 7fa44bfff700 1 -- 172.21.15.49:0/891102258 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4540095f0 con 0x7fa464060280 2023-11-30T21:52:22.404 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.402+0000 7fa469c61700 1 -- 172.21.15.49:0/891102258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa464045ad0 con 0x7fa464060280 2023-11-30T21:52:22.404 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.402+0000 7fa44bfff700 1 --2- 172.21.15.49:0/891102258 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa44c02ffb0 0x7fa44c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:22.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.403+0000 7fa44bfff700 1 -- 172.21.15.49:0/891102258 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa454041050 con 0x7fa464060280 2023-11-30T21:52:22.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.403+0000 7fa462ffd700 1 --2- 172.21.15.49:0/891102258 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa44c02ffb0 0x7fa44c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:22.405 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.403+0000 7fa462ffd700 1 --2- 172.21.15.49:0/891102258 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa44c02ffb0 0x7fa44c032470 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa4580096a0 tx=0x7fa458006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:22.408 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.407+0000 7fa44bfff700 1 -- 172.21.15.49:0/891102258 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa454014420 con 0x7fa464060280 2023-11-30T21:52:22.624 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.623+0000 7fa469c61700 1 -- 172.21.15.49:0/891102258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa464107e80 con 0x7fa464060280 2023-11-30T21:52:22.625 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.624+0000 7fa44bfff700 1 -- 172.21.15.49:0/891102258 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa454010230 con 0x7fa464060280 2023-11-30T21:52:22.625 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:22.625 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:22.626 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.626+0000 7fa469c61700 1 -- 172.21.15.49:0/891102258 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa44c02ffb0 msgr2=0x7fa44c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:22.626 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.626+0000 7fa469c61700 1 --2- 172.21.15.49:0/891102258 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa44c02ffb0 0x7fa44c032470 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fa4580096a0 tx=0x7fa458006b40).stop 2023-11-30T21:52:22.626 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.626+0000 7fa469c61700 1 -- 172.21.15.49:0/891102258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa464060280 msgr2=0x7fa464106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:22.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.626+0000 7fa469c61700 1 --2- 172.21.15.49:0/891102258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa464060280 0x7fa464106500 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fa454013510 tx=0x7fa45400b040).stop 2023-11-30T21:52:22.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.626+0000 7fa469c61700 1 -- 172.21.15.49:0/891102258 shutdown_connections 2023-11-30T21:52:22.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.626+0000 7fa469c61700 1 --2- 172.21.15.49:0/891102258 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fa44c02ffb0 0x7fa44c032470 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:22.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.626+0000 7fa469c61700 1 --2- 172.21.15.49:0/891102258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa464060280 0x7fa464106500 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:22.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.626+0000 7fa469c61700 1 -- 172.21.15.49:0/891102258 >> 172.21.15.49:0/891102258 conn(0x7fa4640f0920 msgr2=0x7fa4640f13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:22.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.626+0000 7fa469c61700 1 -- 172.21.15.49:0/891102258 shutdown_connections 2023-11-30T21:52:22.627 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:22.626+0000 7fa469c61700 1 -- 172.21.15.49:0/891102258 wait complete. 2023-11-30T21:52:22.627 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:23.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:23 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:21.538114+0000 mgr.smithi027.brrjcl (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:23.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:23 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:22.624578+0000 mon.smithi027 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.49:0/891102258' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:24.075 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:24.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:25.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.313+0000 7fbfb534a700 1 -- 172.21.15.49:0/3642715233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb0060280 msgr2=0x7fbfb00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:25.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.313+0000 7fbfb534a700 1 --2- 172.21.15.49:0/3642715233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb0060280 0x7fbfb00f90f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fbfa0009fb0 tx=0x7fbfa0009b30).stop 2023-11-30T21:52:25.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.313+0000 7fbfb534a700 1 -- 172.21.15.49:0/3642715233 shutdown_connections 2023-11-30T21:52:25.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.313+0000 7fbfb534a700 1 --2- 172.21.15.49:0/3642715233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb0060280 0x7fbfb00f90f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:25.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.313+0000 7fbfb534a700 1 -- 172.21.15.49:0/3642715233 >> 172.21.15.49:0/3642715233 conn(0x7fbfb00f0b70 msgr2=0x7fbfb00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:25.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.313+0000 7fbfb534a700 1 -- 172.21.15.49:0/3642715233 shutdown_connections 2023-11-30T21:52:25.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.313+0000 7fbfb534a700 1 -- 172.21.15.49:0/3642715233 wait complete. 2023-11-30T21:52:25.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.314+0000 7fbfb534a700 1 Processor -- start 2023-11-30T21:52:25.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.314+0000 7fbfb534a700 1 -- start start 2023-11-30T21:52:25.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.314+0000 7fbfb534a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb0060280 0x7fbfb01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:25.317 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.314+0000 7fbfb534a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb0106bd0 con 0x7fbfb0060280 2023-11-30T21:52:25.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.314+0000 7fbfaeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb0060280 0x7fbfb01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:25.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.314+0000 7fbfaeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb0060280 0x7fbfb01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:37738/0 (socket says 172.21.15.49:37738) 2023-11-30T21:52:25.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.314+0000 7fbfaeffd700 1 -- 172.21.15.49:0/490606963 learned_addr learned my addr 172.21.15.49:0/490606963 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:25.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.314+0000 7fbfaeffd700 1 -- 172.21.15.49:0/490606963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa0005040 con 0x7fbfb0060280 2023-11-30T21:52:25.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.315+0000 7fbfaeffd700 1 --2- 172.21.15.49:0/490606963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb0060280 0x7fbfb01066e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fbfa0013510 tx=0x7fbfa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:25.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.315+0000 7fbfa77fe700 1 -- 172.21.15.49:0/490606963 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfa0009320 con 0x7fbfb0060280 2023-11-30T21:52:25.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.315+0000 7fbfb534a700 1 -- 172.21.15.49:0/490606963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb0106dd0 con 0x7fbfb0060280 2023-11-30T21:52:25.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.315+0000 7fbfa77fe700 1 -- 172.21.15.49:0/490606963 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfa0013f00 con 0x7fbfb0060280 2023-11-30T21:52:25.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.315+0000 7fbfa77fe700 1 -- 172.21.15.49:0/490606963 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfa0014dc0 con 0x7fbfb0060280 2023-11-30T21:52:25.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.315+0000 7fbfb534a700 1 -- 172.21.15.49:0/490606963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb0107260 con 0x7fbfb0060280 2023-11-30T21:52:25.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.316+0000 7fbfa77fe700 1 -- 172.21.15.49:0/490606963 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbfa001d3d0 con 0x7fbfb0060280 2023-11-30T21:52:25.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.316+0000 7fbfa77fe700 1 --2- 172.21.15.49:0/490606963 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fbf9c02ff60 0x7fbf9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:25.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.316+0000 7fbfa77fe700 1 -- 172.21.15.49:0/490606963 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbfa0041030 con 0x7fbfb0060280 2023-11-30T21:52:25.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.316+0000 7fbfb534a700 1 -- 172.21.15.49:0/490606963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb0045ad0 con 0x7fbfb0060280 2023-11-30T21:52:25.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.316+0000 7fbfae7fc700 1 --2- 172.21.15.49:0/490606963 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fbf9c02ff60 0x7fbf9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:25.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.316+0000 7fbfae7fc700 1 --2- 172.21.15.49:0/490606963 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fbf9c02ff60 0x7fbf9c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fbf980096a0 tx=0x7fbf98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:25.322 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.321+0000 7fbfa77fe700 1 -- 172.21.15.49:0/490606963 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbfa0026490 con 0x7fbfb0060280 2023-11-30T21:52:25.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:23.538390+0000 mgr.smithi027.brrjcl (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:25.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.538+0000 7fbfb534a700 1 -- 172.21.15.49:0/490606963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbfb01080b0 con 0x7fbfb0060280 2023-11-30T21:52:25.540 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.539+0000 7fbfa77fe700 1 -- 172.21.15.49:0/490606963 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbfa002b020 con 0x7fbfb0060280 2023-11-30T21:52:25.540 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:25.540 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:25.541 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.540+0000 7fbfb534a700 1 -- 172.21.15.49:0/490606963 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fbf9c02ff60 msgr2=0x7fbf9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:25.541 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.540+0000 7fbfb534a700 1 --2- 172.21.15.49:0/490606963 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fbf9c02ff60 0x7fbf9c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fbf980096a0 tx=0x7fbf98006b40).stop 2023-11-30T21:52:25.541 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.540+0000 7fbfb534a700 1 -- 172.21.15.49:0/490606963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb0060280 msgr2=0x7fbfb01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:25.541 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.540+0000 7fbfb534a700 1 --2- 172.21.15.49:0/490606963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb0060280 0x7fbfb01066e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fbfa0013510 tx=0x7fbfa000b040).stop 2023-11-30T21:52:25.541 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.541+0000 7fbfb534a700 1 -- 172.21.15.49:0/490606963 shutdown_connections 2023-11-30T21:52:25.542 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.541+0000 7fbfb534a700 1 --2- 172.21.15.49:0/490606963 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fbf9c02ff60 0x7fbf9c032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:25.542 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.541+0000 7fbfb534a700 1 --2- 172.21.15.49:0/490606963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfb0060280 0x7fbfb01066e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:25.542 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.541+0000 7fbfb534a700 1 -- 172.21.15.49:0/490606963 >> 172.21.15.49:0/490606963 conn(0x7fbfb00f0b70 msgr2=0x7fbfb00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:25.542 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.541+0000 7fbfb534a700 1 -- 172.21.15.49:0/490606963 shutdown_connections 2023-11-30T21:52:25.542 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:25.541+0000 7fbfb534a700 1 -- 172.21.15.49:0/490606963 wait complete. 2023-11-30T21:52:25.542 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:26.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:26 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:25.539664+0000 mon.smithi027 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.49:0/490606963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:26.938 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:26.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:27.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:27 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:25.538660+0000 mgr.smithi027.brrjcl (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:28.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.156+0000 7f72294cd700 1 -- 172.21.15.49:0/3641969342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72240f62a0 msgr2=0x7f72240f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:28.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.156+0000 7f72294cd700 1 --2- 172.21.15.49:0/3641969342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72240f62a0 0x7f72240f66c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f7214009fb0 tx=0x7f7214009b30).stop 2023-11-30T21:52:28.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.156+0000 7f72294cd700 1 -- 172.21.15.49:0/3641969342 shutdown_connections 2023-11-30T21:52:28.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.156+0000 7f72294cd700 1 --2- 172.21.15.49:0/3641969342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72240f62a0 0x7f72240f66c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:28.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.156+0000 7f72294cd700 1 -- 172.21.15.49:0/3641969342 >> 172.21.15.49:0/3641969342 conn(0x7f72240f1aa0 msgr2=0x7f72240f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:28.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.156+0000 7f72294cd700 1 -- 172.21.15.49:0/3641969342 shutdown_connections 2023-11-30T21:52:28.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.156+0000 7f72294cd700 1 -- 172.21.15.49:0/3641969342 wait complete. 2023-11-30T21:52:28.159 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.157+0000 7f72294cd700 1 Processor -- start 2023-11-30T21:52:28.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.157+0000 7f72294cd700 1 -- start start 2023-11-30T21:52:28.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.157+0000 7f72294cd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72240f62a0 0x7f72241064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:28.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.157+0000 7f72294cd700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72241069a0 con 0x7f72240f62a0 2023-11-30T21:52:28.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.158+0000 7f7223fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72240f62a0 0x7f72241064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:28.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.158+0000 7f7223fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72240f62a0 0x7f72241064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:37750/0 (socket says 172.21.15.49:37750) 2023-11-30T21:52:28.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.158+0000 7f7223fff700 1 -- 172.21.15.49:0/40972473 learned_addr learned my addr 172.21.15.49:0/40972473 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:28.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.158+0000 7f7223fff700 1 -- 172.21.15.49:0/40972473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7214005040 con 0x7f72240f62a0 2023-11-30T21:52:28.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.158+0000 7f7223fff700 1 --2- 172.21.15.49:0/40972473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72240f62a0 0x7f72241064b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f7214013510 tx=0x7f721400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:28.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.158+0000 7f7220ff9700 1 -- 172.21.15.49:0/40972473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7214009320 con 0x7f72240f62a0 2023-11-30T21:52:28.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.158+0000 7f72294cd700 1 -- 172.21.15.49:0/40972473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7224106ba0 con 0x7f72240f62a0 2023-11-30T21:52:28.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.158+0000 7f7220ff9700 1 -- 172.21.15.49:0/40972473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7214013f00 con 0x7f72240f62a0 2023-11-30T21:52:28.160 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.158+0000 7f7220ff9700 1 -- 172.21.15.49:0/40972473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7214014dc0 con 0x7f72240f62a0 2023-11-30T21:52:28.161 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.158+0000 7f72294cd700 1 -- 172.21.15.49:0/40972473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7224107030 con 0x7f72240f62a0 2023-11-30T21:52:28.161 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.159+0000 7f7220ff9700 1 -- 172.21.15.49:0/40972473 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f721401d3d0 con 0x7f72240f62a0 2023-11-30T21:52:28.161 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.159+0000 7f7220ff9700 1 --2- 172.21.15.49:0/40972473 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f721002ffb0 0x7f7210032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:28.162 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.159+0000 7f7220ff9700 1 -- 172.21.15.49:0/40972473 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72140410c0 con 0x7f72240f62a0 2023-11-30T21:52:28.162 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.159+0000 7f72294cd700 1 -- 172.21.15.49:0/40972473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7224059100 con 0x7f72240f62a0 2023-11-30T21:52:28.162 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.159+0000 7f72237fe700 1 --2- 172.21.15.49:0/40972473 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f721002ffb0 0x7f7210032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:28.162 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.160+0000 7f72237fe700 1 --2- 172.21.15.49:0/40972473 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f721002ffb0 0x7f7210032470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f720c0096a0 tx=0x7f720c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:28.165 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.164+0000 7f7220ff9700 1 -- 172.21.15.49:0/40972473 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7214012900 con 0x7f72240f62a0 2023-11-30T21:52:28.383 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.382+0000 7f72294cd700 1 -- 172.21.15.49:0/40972473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72240fbce0 con 0x7f72240f62a0 2023-11-30T21:52:28.383 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.383+0000 7f7220ff9700 1 -- 172.21.15.49:0/40972473 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f721402b020 con 0x7f72240f62a0 2023-11-30T21:52:28.383 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:28.384 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:28.385 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.384+0000 7f72294cd700 1 -- 172.21.15.49:0/40972473 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f721002ffb0 msgr2=0x7f7210032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:28.385 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.384+0000 7f72294cd700 1 --2- 172.21.15.49:0/40972473 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f721002ffb0 0x7f7210032470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f720c0096a0 tx=0x7f720c006b40).stop 2023-11-30T21:52:28.385 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.384+0000 7f72294cd700 1 -- 172.21.15.49:0/40972473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72240f62a0 msgr2=0x7f72241064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:28.385 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.384+0000 7f72294cd700 1 --2- 172.21.15.49:0/40972473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72240f62a0 0x7f72241064b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f7214013510 tx=0x7f721400b040).stop 2023-11-30T21:52:28.385 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.384+0000 7f72294cd700 1 -- 172.21.15.49:0/40972473 shutdown_connections 2023-11-30T21:52:28.385 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.384+0000 7f72294cd700 1 --2- 172.21.15.49:0/40972473 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f721002ffb0 0x7f7210032470 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:28.385 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.384+0000 7f72294cd700 1 --2- 172.21.15.49:0/40972473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72240f62a0 0x7f72241064b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:28.386 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.384+0000 7f72294cd700 1 -- 172.21.15.49:0/40972473 >> 172.21.15.49:0/40972473 conn(0x7f72240f1aa0 msgr2=0x7f72240f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:28.386 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.384+0000 7f72294cd700 1 -- 172.21.15.49:0/40972473 shutdown_connections 2023-11-30T21:52:28.386 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:28.384+0000 7f72294cd700 1 -- 172.21.15.49:0/40972473 wait complete. 2023-11-30T21:52:28.386 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:29.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:27.538929+0000 mgr.smithi027.brrjcl (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:29.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:28.383282+0000 mon.smithi027 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.49:0/40972473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:29.822 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:29.822 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:31.068 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.067+0000 7f0a75525700 1 -- 172.21.15.49:0/2865793155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a700f51a0 msgr2=0x7f0a700f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:31.068 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.067+0000 7f0a75525700 1 --2- 172.21.15.49:0/2865793155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a700f51a0 0x7f0a700f55c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f0a60009fb0 tx=0x7f0a60009b30).stop 2023-11-30T21:52:31.068 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.067+0000 7f0a75525700 1 -- 172.21.15.49:0/2865793155 shutdown_connections 2023-11-30T21:52:31.068 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.067+0000 7f0a75525700 1 --2- 172.21.15.49:0/2865793155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a700f51a0 0x7f0a700f55c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:31.068 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.067+0000 7f0a75525700 1 -- 172.21.15.49:0/2865793155 >> 172.21.15.49:0/2865793155 conn(0x7f0a700f09a0 msgr2=0x7f0a700f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:31.069 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.067+0000 7f0a75525700 1 -- 172.21.15.49:0/2865793155 shutdown_connections 2023-11-30T21:52:31.069 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.067+0000 7f0a75525700 1 -- 172.21.15.49:0/2865793155 wait complete. 2023-11-30T21:52:31.069 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.067+0000 7f0a75525700 1 Processor -- start 2023-11-30T21:52:31.069 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.068+0000 7f0a75525700 1 -- start start 2023-11-30T21:52:31.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.068+0000 7f0a75525700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a700f51a0 0x7f0a701091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:31.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.068+0000 7f0a75525700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a70060240 con 0x7f0a700f51a0 2023-11-30T21:52:31.070 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.068+0000 7f0a6ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a700f51a0 0x7f0a701091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:31.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.068+0000 7f0a6ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a700f51a0 0x7f0a701091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:46966/0 (socket says 172.21.15.49:46966) 2023-11-30T21:52:31.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.068+0000 7f0a6ffff700 1 -- 172.21.15.49:0/4115517678 learned_addr learned my addr 172.21.15.49:0/4115517678 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:31.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.068+0000 7f0a6ffff700 1 -- 172.21.15.49:0/4115517678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a60005040 con 0x7f0a700f51a0 2023-11-30T21:52:31.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.069+0000 7f0a6ffff700 1 --2- 172.21.15.49:0/4115517678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a700f51a0 0x7f0a701091e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f0a60013510 tx=0x7f0a6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:31.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.069+0000 7f0a6cff9700 1 -- 172.21.15.49:0/4115517678 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a60009320 con 0x7f0a700f51a0 2023-11-30T21:52:31.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.069+0000 7f0a75525700 1 -- 172.21.15.49:0/4115517678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a70060440 con 0x7f0a700f51a0 2023-11-30T21:52:31.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.069+0000 7f0a6cff9700 1 -- 172.21.15.49:0/4115517678 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a60013f00 con 0x7f0a700f51a0 2023-11-30T21:52:31.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.069+0000 7f0a6cff9700 1 -- 172.21.15.49:0/4115517678 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a60014dc0 con 0x7f0a700f51a0 2023-11-30T21:52:31.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.069+0000 7f0a75525700 1 -- 172.21.15.49:0/4115517678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a70109890 con 0x7f0a700f51a0 2023-11-30T21:52:31.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.070+0000 7f0a6cff9700 1 -- 172.21.15.49:0/4115517678 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0a600263a0 con 0x7f0a700f51a0 2023-11-30T21:52:31.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.070+0000 7f0a6cff9700 1 --2- 172.21.15.49:0/4115517678 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0a5802ffb0 0x7f0a58032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:31.071 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.070+0000 7f0a6cff9700 1 -- 172.21.15.49:0/4115517678 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a60041000 con 0x7f0a700f51a0 2023-11-30T21:52:31.072 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.070+0000 7f0a75525700 1 -- 172.21.15.49:0/4115517678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a70045ad0 con 0x7f0a700f51a0 2023-11-30T21:52:31.072 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.070+0000 7f0a6f7fe700 1 --2- 172.21.15.49:0/4115517678 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0a5802ffb0 0x7f0a58032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:31.072 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.070+0000 7f0a6f7fe700 1 --2- 172.21.15.49:0/4115517678 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0a5802ffb0 0x7f0a58032470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f0a640096a0 tx=0x7f0a64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:31.075 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.075+0000 7f0a6cff9700 1 -- 172.21.15.49:0/4115517678 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a60012920 con 0x7f0a700f51a0 2023-11-30T21:52:31.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.292+0000 7f0a75525700 1 -- 172.21.15.49:0/4115517678 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a70109f70 con 0x7f0a700f51a0 2023-11-30T21:52:31.293 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.292+0000 7f0a6cff9700 1 -- 172.21.15.49:0/4115517678 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0a6002b020 con 0x7f0a700f51a0 2023-11-30T21:52:31.293 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:31.294 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:31.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.294+0000 7f0a75525700 1 -- 172.21.15.49:0/4115517678 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0a5802ffb0 msgr2=0x7f0a58032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:31.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.294+0000 7f0a75525700 1 --2- 172.21.15.49:0/4115517678 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0a5802ffb0 0x7f0a58032470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f0a640096a0 tx=0x7f0a64006b40).stop 2023-11-30T21:52:31.295 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.294+0000 7f0a75525700 1 -- 172.21.15.49:0/4115517678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a700f51a0 msgr2=0x7f0a701091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:31.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.294+0000 7f0a75525700 1 --2- 172.21.15.49:0/4115517678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a700f51a0 0x7f0a701091e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f0a60013510 tx=0x7f0a6000b040).stop 2023-11-30T21:52:31.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.294+0000 7f0a75525700 1 -- 172.21.15.49:0/4115517678 shutdown_connections 2023-11-30T21:52:31.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.294+0000 7f0a75525700 1 --2- 172.21.15.49:0/4115517678 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f0a5802ffb0 0x7f0a58032470 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:31.296 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.294+0000 7f0a75525700 1 --2- 172.21.15.49:0/4115517678 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a700f51a0 0x7f0a701091e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:31.297 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.294+0000 7f0a75525700 1 -- 172.21.15.49:0/4115517678 >> 172.21.15.49:0/4115517678 conn(0x7f0a700f09a0 msgr2=0x7f0a700f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:31.297 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.294+0000 7f0a75525700 1 -- 172.21.15.49:0/4115517678 shutdown_connections 2023-11-30T21:52:31.297 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:31.294+0000 7f0a75525700 1 -- 172.21.15.49:0/4115517678 wait complete. 2023-11-30T21:52:31.297 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:31.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:29.539277+0000 mgr.smithi027.brrjcl (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:32.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:32 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:31.292813+0000 mon.smithi027 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.49:0/4115517678' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:32.724 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:32.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:33.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:33 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:31.539659+0000 mgr.smithi027.brrjcl (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:34.030 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.025+0000 7f49165b6700 1 -- 172.21.15.49:0/1630579063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49100f6510 msgr2=0x7f49100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:34.030 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.025+0000 7f49165b6700 1 --2- 172.21.15.49:0/1630579063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49100f6510 0x7f49100f6930 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f4900009fb0 tx=0x7f4900009b30).stop 2023-11-30T21:52:34.030 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.025+0000 7f49165b6700 1 -- 172.21.15.49:0/1630579063 shutdown_connections 2023-11-30T21:52:34.030 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.025+0000 7f49165b6700 1 --2- 172.21.15.49:0/1630579063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49100f6510 0x7f49100f6930 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:34.031 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.025+0000 7f49165b6700 1 -- 172.21.15.49:0/1630579063 >> 172.21.15.49:0/1630579063 conn(0x7f49100f1c70 msgr2=0x7f49100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:34.031 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.025+0000 7f49165b6700 1 -- 172.21.15.49:0/1630579063 shutdown_connections 2023-11-30T21:52:34.031 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.025+0000 7f49165b6700 1 -- 172.21.15.49:0/1630579063 wait complete. 2023-11-30T21:52:34.031 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.026+0000 7f49165b6700 1 Processor -- start 2023-11-30T21:52:34.031 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.026+0000 7f49165b6700 1 -- start start 2023-11-30T21:52:34.031 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.026+0000 7f49165b6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49100f6510 0x7f4910106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:34.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.026+0000 7f49165b6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4910106c20 con 0x7f49100f6510 2023-11-30T21:52:34.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.026+0000 7f490ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49100f6510 0x7f4910106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:34.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.026+0000 7f490ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49100f6510 0x7f4910106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:46986/0 (socket says 172.21.15.49:46986) 2023-11-30T21:52:34.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.026+0000 7f490ffff700 1 -- 172.21.15.49:0/2138409286 learned_addr learned my addr 172.21.15.49:0/2138409286 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:34.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.027+0000 7f490ffff700 1 -- 172.21.15.49:0/2138409286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4900005040 con 0x7f49100f6510 2023-11-30T21:52:34.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.027+0000 7f490ffff700 1 --2- 172.21.15.49:0/2138409286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49100f6510 0x7f4910106730 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f4900014000 tx=0x7f490000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:34.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.027+0000 7f490cff9700 1 -- 172.21.15.49:0/2138409286 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49000095f0 con 0x7f49100f6510 2023-11-30T21:52:34.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.027+0000 7f49165b6700 1 -- 172.21.15.49:0/2138409286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4910106e20 con 0x7f49100f6510 2023-11-30T21:52:34.032 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.027+0000 7f490cff9700 1 -- 172.21.15.49:0/2138409286 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4900007e70 con 0x7f49100f6510 2023-11-30T21:52:34.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.027+0000 7f490cff9700 1 -- 172.21.15.49:0/2138409286 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4900014ea0 con 0x7f49100f6510 2023-11-30T21:52:34.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.027+0000 7f49165b6700 1 -- 172.21.15.49:0/2138409286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49101072b0 con 0x7f49100f6510 2023-11-30T21:52:34.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.028+0000 7f490cff9700 1 -- 172.21.15.49:0/2138409286 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4900025070 con 0x7f49100f6510 2023-11-30T21:52:34.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.028+0000 7f49165b6700 1 -- 172.21.15.49:0/2138409286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4910045ad0 con 0x7f49100f6510 2023-11-30T21:52:34.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.028+0000 7f490cff9700 1 --2- 172.21.15.49:0/2138409286 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f48f802ff60 0x7f48f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:34.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.028+0000 7f490cff9700 1 -- 172.21.15.49:0/2138409286 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f49000409f0 con 0x7f49100f6510 2023-11-30T21:52:34.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.028+0000 7f490f7fe700 1 --2- 172.21.15.49:0/2138409286 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f48f802ff60 0x7f48f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:34.033 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.029+0000 7f490f7fe700 1 --2- 172.21.15.49:0/2138409286 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f48f802ff60 0x7f48f8032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f49040096a0 tx=0x7f4904006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:34.034 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.033+0000 7f490cff9700 1 -- 172.21.15.49:0/2138409286 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f490000f4e0 con 0x7f49100f6510 2023-11-30T21:52:34.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.248+0000 7f49165b6700 1 -- 172.21.15.49:0/2138409286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49100fbf60 con 0x7f49100f6510 2023-11-30T21:52:34.250 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.249+0000 7f490cff9700 1 -- 172.21.15.49:0/2138409286 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f490000f300 con 0x7f49100f6510 2023-11-30T21:52:34.250 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:34.251 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:34.252 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.250+0000 7f49165b6700 1 -- 172.21.15.49:0/2138409286 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f48f802ff60 msgr2=0x7f48f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:34.252 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.250+0000 7f49165b6700 1 --2- 172.21.15.49:0/2138409286 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f48f802ff60 0x7f48f8032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f49040096a0 tx=0x7f4904006b40).stop 2023-11-30T21:52:34.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.250+0000 7f49165b6700 1 -- 172.21.15.49:0/2138409286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49100f6510 msgr2=0x7f4910106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:34.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.250+0000 7f49165b6700 1 --2- 172.21.15.49:0/2138409286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49100f6510 0x7f4910106730 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f4900014000 tx=0x7f490000b040).stop 2023-11-30T21:52:34.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.250+0000 7f49165b6700 1 -- 172.21.15.49:0/2138409286 shutdown_connections 2023-11-30T21:52:34.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.250+0000 7f49165b6700 1 --2- 172.21.15.49:0/2138409286 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f48f802ff60 0x7f48f8032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:34.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.250+0000 7f49165b6700 1 --2- 172.21.15.49:0/2138409286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49100f6510 0x7f4910106730 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:34.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.250+0000 7f49165b6700 1 -- 172.21.15.49:0/2138409286 >> 172.21.15.49:0/2138409286 conn(0x7f49100f1c70 msgr2=0x7f49100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:34.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.250+0000 7f49165b6700 1 -- 172.21.15.49:0/2138409286 shutdown_connections 2023-11-30T21:52:34.253 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:34.250+0000 7f49165b6700 1 -- 172.21.15.49:0/2138409286 wait complete. 2023-11-30T21:52:34.253 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:35.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:35 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:33.539978+0000 mgr.smithi027.brrjcl (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:35.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:35 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:34.250291+0000 mon.smithi027 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.49:0/2138409286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:35.652 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:35.652 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:36.874 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.872+0000 7fac6f5b2700 1 -- 172.21.15.49:0/177549337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680f5410 msgr2=0x7fac680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:36.874 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.872+0000 7fac6f5b2700 1 --2- 172.21.15.49:0/177549337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680f5410 0x7fac680f5830 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fac64009fb0 tx=0x7fac64009b30).stop 2023-11-30T21:52:36.875 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.872+0000 7fac6f5b2700 1 -- 172.21.15.49:0/177549337 shutdown_connections 2023-11-30T21:52:36.875 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.872+0000 7fac6f5b2700 1 --2- 172.21.15.49:0/177549337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680f5410 0x7fac680f5830 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:36.875 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.872+0000 7fac6f5b2700 1 -- 172.21.15.49:0/177549337 >> 172.21.15.49:0/177549337 conn(0x7fac680f0b70 msgr2=0x7fac680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:36.875 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.872+0000 7fac6f5b2700 1 -- 172.21.15.49:0/177549337 shutdown_connections 2023-11-30T21:52:36.875 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.872+0000 7fac6f5b2700 1 -- 172.21.15.49:0/177549337 wait complete. 2023-11-30T21:52:36.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.873+0000 7fac6f5b2700 1 Processor -- start 2023-11-30T21:52:36.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.873+0000 7fac6f5b2700 1 -- start start 2023-11-30T21:52:36.876 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.873+0000 7fac6f5b2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680f5410 0x7fac681093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:36.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.873+0000 7fac6f5b2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac68060220 con 0x7fac680f5410 2023-11-30T21:52:36.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.874+0000 7fac6d34e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680f5410 0x7fac681093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:36.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.874+0000 7fac6d34e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680f5410 0x7fac681093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:46998/0 (socket says 172.21.15.49:46998) 2023-11-30T21:52:36.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.874+0000 7fac6d34e700 1 -- 172.21.15.49:0/453919911 learned_addr learned my addr 172.21.15.49:0/453919911 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:36.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.874+0000 7fac6d34e700 1 -- 172.21.15.49:0/453919911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac64005040 con 0x7fac680f5410 2023-11-30T21:52:36.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.874+0000 7fac6d34e700 1 --2- 172.21.15.49:0/453919911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680f5410 0x7fac681093f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fac64013510 tx=0x7fac6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:36.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.874+0000 7fac59ffb700 1 -- 172.21.15.49:0/453919911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac64009320 con 0x7fac680f5410 2023-11-30T21:52:36.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.874+0000 7fac6f5b2700 1 -- 172.21.15.49:0/453919911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac68060420 con 0x7fac680f5410 2023-11-30T21:52:36.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.874+0000 7fac59ffb700 1 -- 172.21.15.49:0/453919911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac64013f00 con 0x7fac680f5410 2023-11-30T21:52:36.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.874+0000 7fac59ffb700 1 -- 172.21.15.49:0/453919911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac64014d90 con 0x7fac680f5410 2023-11-30T21:52:36.877 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.874+0000 7fac6f5b2700 1 -- 172.21.15.49:0/453919911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac68109aa0 con 0x7fac680f5410 2023-11-30T21:52:36.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.875+0000 7fac59ffb700 1 -- 172.21.15.49:0/453919911 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac640095f0 con 0x7fac680f5410 2023-11-30T21:52:36.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.875+0000 7fac6f5b2700 1 -- 172.21.15.49:0/453919911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac68059100 con 0x7fac680f5410 2023-11-30T21:52:36.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.875+0000 7fac59ffb700 1 --2- 172.21.15.49:0/453919911 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fac5402ff60 0x7fac54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:36.878 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.875+0000 7fac59ffb700 1 -- 172.21.15.49:0/453919911 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac64041000 con 0x7fac680f5410 2023-11-30T21:52:36.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.876+0000 7fac6cb4d700 1 --2- 172.21.15.49:0/453919911 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fac5402ff60 0x7fac54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:36.879 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.876+0000 7fac6cb4d700 1 --2- 172.21.15.49:0/453919911 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fac5402ff60 0x7fac54032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fac5c0096a0 tx=0x7fac5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:36.881 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:36.880+0000 7fac59ffb700 1 -- 172.21.15.49:0/453919911 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac6401d7f0 con 0x7fac680f5410 2023-11-30T21:52:37.100 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:37.098+0000 7fac6f5b2700 1 -- 172.21.15.49:0/453919911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac68045ad0 con 0x7fac680f5410 2023-11-30T21:52:37.101 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:37.099+0000 7fac59ffb700 1 -- 172.21.15.49:0/453919911 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac64010230 con 0x7fac680f5410 2023-11-30T21:52:37.101 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:37.101 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:37.103 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:37.101+0000 7fac6f5b2700 1 -- 172.21.15.49:0/453919911 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fac5402ff60 msgr2=0x7fac54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:37.103 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:37.101+0000 7fac6f5b2700 1 --2- 172.21.15.49:0/453919911 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fac5402ff60 0x7fac54032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fac5c0096a0 tx=0x7fac5c006b40).stop 2023-11-30T21:52:37.103 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:37.101+0000 7fac6f5b2700 1 -- 172.21.15.49:0/453919911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680f5410 msgr2=0x7fac681093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:37.103 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:37.101+0000 7fac6f5b2700 1 --2- 172.21.15.49:0/453919911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680f5410 0x7fac681093f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fac64013510 tx=0x7fac6400b040).stop 2023-11-30T21:52:37.103 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:37.101+0000 7fac6f5b2700 1 -- 172.21.15.49:0/453919911 shutdown_connections 2023-11-30T21:52:37.103 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:37.101+0000 7fac6f5b2700 1 --2- 172.21.15.49:0/453919911 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fac5402ff60 0x7fac54032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:37.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:37.101+0000 7fac6f5b2700 1 --2- 172.21.15.49:0/453919911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac680f5410 0x7fac681093f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:37.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:37.101+0000 7fac6f5b2700 1 -- 172.21.15.49:0/453919911 >> 172.21.15.49:0/453919911 conn(0x7fac680f0b70 msgr2=0x7fac680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:37.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:37.101+0000 7fac6f5b2700 1 -- 172.21.15.49:0/453919911 shutdown_connections 2023-11-30T21:52:37.104 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:37.101+0000 7fac6f5b2700 1 -- 172.21.15.49:0/453919911 wait complete. 2023-11-30T21:52:37.104 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:37.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:37 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:35.540263+0000 mgr.smithi027.brrjcl (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:37.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:37 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:37.100455+0000 mon.smithi027 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.49:0/453919911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:38.494 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:38.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:39.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:39 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:37.540654+0000 mgr.smithi027.brrjcl (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:39.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.796+0000 7fd9bad75700 1 -- 172.21.15.49:0/4177830927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b40f5410 msgr2=0x7fd9b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:39.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.796+0000 7fd9bad75700 1 --2- 172.21.15.49:0/4177830927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b40f5410 0x7fd9b40f5830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fd9a8009fb0 tx=0x7fd9a8009b30).stop 2023-11-30T21:52:39.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.797+0000 7fd9bad75700 1 -- 172.21.15.49:0/4177830927 shutdown_connections 2023-11-30T21:52:39.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.797+0000 7fd9bad75700 1 --2- 172.21.15.49:0/4177830927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b40f5410 0x7fd9b40f5830 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:39.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.797+0000 7fd9bad75700 1 -- 172.21.15.49:0/4177830927 >> 172.21.15.49:0/4177830927 conn(0x7fd9b40f0b70 msgr2=0x7fd9b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:39.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.797+0000 7fd9bad75700 1 -- 172.21.15.49:0/4177830927 shutdown_connections 2023-11-30T21:52:39.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.797+0000 7fd9bad75700 1 -- 172.21.15.49:0/4177830927 wait complete. 2023-11-30T21:52:39.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.797+0000 7fd9bad75700 1 Processor -- start 2023-11-30T21:52:39.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.797+0000 7fd9bad75700 1 -- start start 2023-11-30T21:52:39.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.797+0000 7fd9bad75700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b40f5410 0x7fd9b40fb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:39.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.797+0000 7fd9bad75700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b40f9a00 con 0x7fd9b40f5410 2023-11-30T21:52:39.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.798+0000 7fd9b8b11700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b40f5410 0x7fd9b40fb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:39.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.798+0000 7fd9b8b11700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b40f5410 0x7fd9b40fb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:41472/0 (socket says 172.21.15.49:41472) 2023-11-30T21:52:39.801 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.798+0000 7fd9b8b11700 1 -- 172.21.15.49:0/822426296 learned_addr learned my addr 172.21.15.49:0/822426296 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:39.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.798+0000 7fd9b8b11700 1 -- 172.21.15.49:0/822426296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a8005040 con 0x7fd9b40f5410 2023-11-30T21:52:39.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.798+0000 7fd9b8b11700 1 --2- 172.21.15.49:0/822426296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b40f5410 0x7fd9b40fb580 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fd9a8013510 tx=0x7fd9a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:39.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.798+0000 7fd9b17fa700 1 -- 172.21.15.49:0/822426296 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9a8009320 con 0x7fd9b40f5410 2023-11-30T21:52:39.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.798+0000 7fd9bad75700 1 -- 172.21.15.49:0/822426296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b40f9ba0 con 0x7fd9b40f5410 2023-11-30T21:52:39.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.798+0000 7fd9b17fa700 1 -- 172.21.15.49:0/822426296 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9a8013f00 con 0x7fd9b40f5410 2023-11-30T21:52:39.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.798+0000 7fd9b17fa700 1 -- 172.21.15.49:0/822426296 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9a8014d90 con 0x7fd9b40f5410 2023-11-30T21:52:39.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.798+0000 7fd9bad75700 1 -- 172.21.15.49:0/822426296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b40fa030 con 0x7fd9b40f5410 2023-11-30T21:52:39.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.799+0000 7fd9b17fa700 1 -- 172.21.15.49:0/822426296 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9a80095f0 con 0x7fd9b40f5410 2023-11-30T21:52:39.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.799+0000 7fd9bad75700 1 -- 172.21.15.49:0/822426296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b4059100 con 0x7fd9b40f5410 2023-11-30T21:52:39.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.799+0000 7fd9b17fa700 1 --2- 172.21.15.49:0/822426296 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd9a402ff60 0x7fd9a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:39.802 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.799+0000 7fd9b17fa700 1 -- 172.21.15.49:0/822426296 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9a80410c0 con 0x7fd9b40f5410 2023-11-30T21:52:39.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.800+0000 7fd9b3fff700 1 --2- 172.21.15.49:0/822426296 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd9a402ff60 0x7fd9a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:39.803 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.800+0000 7fd9b3fff700 1 --2- 172.21.15.49:0/822426296 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd9a402ff60 0x7fd9a4032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fd9a00096a0 tx=0x7fd9a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:39.806 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:39.804+0000 7fd9b17fa700 1 -- 172.21.15.49:0/822426296 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9a8014420 con 0x7fd9b40f5410 2023-11-30T21:52:40.022 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:40.021+0000 7fd9bad75700 1 -- 172.21.15.49:0/822426296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9b40fbf90 con 0x7fd9b40f5410 2023-11-30T21:52:40.023 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:40.022+0000 7fd9b17fa700 1 -- 172.21.15.49:0/822426296 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd9a8010230 con 0x7fd9b40f5410 2023-11-30T21:52:40.024 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:40.024 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:40.025 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:40.023+0000 7fd9bad75700 1 -- 172.21.15.49:0/822426296 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd9a402ff60 msgr2=0x7fd9a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:40.025 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:40.023+0000 7fd9bad75700 1 --2- 172.21.15.49:0/822426296 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd9a402ff60 0x7fd9a4032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fd9a00096a0 tx=0x7fd9a0006b40).stop 2023-11-30T21:52:40.025 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:40.023+0000 7fd9bad75700 1 -- 172.21.15.49:0/822426296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b40f5410 msgr2=0x7fd9b40fb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:40.025 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:40.023+0000 7fd9bad75700 1 --2- 172.21.15.49:0/822426296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b40f5410 0x7fd9b40fb580 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fd9a8013510 tx=0x7fd9a800b040).stop 2023-11-30T21:52:40.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:40.023+0000 7fd9bad75700 1 -- 172.21.15.49:0/822426296 shutdown_connections 2023-11-30T21:52:40.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:40.023+0000 7fd9bad75700 1 --2- 172.21.15.49:0/822426296 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fd9a402ff60 0x7fd9a4032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:40.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:40.023+0000 7fd9bad75700 1 --2- 172.21.15.49:0/822426296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9b40f5410 0x7fd9b40fb580 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:40.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:40.023+0000 7fd9bad75700 1 -- 172.21.15.49:0/822426296 >> 172.21.15.49:0/822426296 conn(0x7fd9b40f0b70 msgr2=0x7fd9b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:40.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:40.024+0000 7fd9bad75700 1 -- 172.21.15.49:0/822426296 shutdown_connections 2023-11-30T21:52:40.026 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:40.024+0000 7fd9bad75700 1 -- 172.21.15.49:0/822426296 wait complete. 2023-11-30T21:52:40.026 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:40.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:40 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:40.023191+0000 mon.smithi027 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.49:0/822426296' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:41.410 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:41.411 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:41.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:41 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:39.541030+0000 mgr.smithi027.brrjcl (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:42.613 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.611+0000 7f8f71c4d700 1 -- 172.21.15.49:0/3454793793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f6c060280 msgr2=0x7f8f6c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:42.614 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.611+0000 7f8f71c4d700 1 --2- 172.21.15.49:0/3454793793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f6c060280 0x7f8f6c0f90f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f8f5c009fb0 tx=0x7f8f5c009b30).stop 2023-11-30T21:52:42.614 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.611+0000 7f8f71c4d700 1 -- 172.21.15.49:0/3454793793 shutdown_connections 2023-11-30T21:52:42.614 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.611+0000 7f8f71c4d700 1 --2- 172.21.15.49:0/3454793793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f6c060280 0x7f8f6c0f90f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:42.614 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.611+0000 7f8f71c4d700 1 -- 172.21.15.49:0/3454793793 >> 172.21.15.49:0/3454793793 conn(0x7f8f6c0f0b70 msgr2=0x7f8f6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:42.614 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.612+0000 7f8f71c4d700 1 -- 172.21.15.49:0/3454793793 shutdown_connections 2023-11-30T21:52:42.614 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.612+0000 7f8f71c4d700 1 -- 172.21.15.49:0/3454793793 wait complete. 2023-11-30T21:52:42.614 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.612+0000 7f8f71c4d700 1 Processor -- start 2023-11-30T21:52:42.616 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.612+0000 7f8f71c4d700 1 -- start start 2023-11-30T21:52:42.616 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.612+0000 7f8f71c4d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f6c060280 0x7f8f6c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:42.616 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.612+0000 7f8f71c4d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c106bd0 con 0x7f8f6c060280 2023-11-30T21:52:42.616 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.613+0000 7f8f6b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f6c060280 0x7f8f6c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:42.616 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.613+0000 7f8f6b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f6c060280 0x7f8f6c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:41484/0 (socket says 172.21.15.49:41484) 2023-11-30T21:52:42.616 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.613+0000 7f8f6b7fe700 1 -- 172.21.15.49:0/4290209754 learned_addr learned my addr 172.21.15.49:0/4290209754 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:42.616 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.613+0000 7f8f6b7fe700 1 -- 172.21.15.49:0/4290209754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c005040 con 0x7f8f6c060280 2023-11-30T21:52:42.617 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.613+0000 7f8f6b7fe700 1 --2- 172.21.15.49:0/4290209754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f6c060280 0x7f8f6c1066e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f8f5c013510 tx=0x7f8f5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:42.617 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.613+0000 7f8f53fff700 1 -- 172.21.15.49:0/4290209754 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f5c009320 con 0x7f8f6c060280 2023-11-30T21:52:42.617 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.613+0000 7f8f71c4d700 1 -- 172.21.15.49:0/4290209754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6c106dd0 con 0x7f8f6c060280 2023-11-30T21:52:42.617 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.613+0000 7f8f53fff700 1 -- 172.21.15.49:0/4290209754 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f5c013f00 con 0x7f8f6c060280 2023-11-30T21:52:42.617 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.613+0000 7f8f53fff700 1 -- 172.21.15.49:0/4290209754 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f5c014dc0 con 0x7f8f6c060280 2023-11-30T21:52:42.617 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.613+0000 7f8f71c4d700 1 -- 172.21.15.49:0/4290209754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6c107260 con 0x7f8f6c060280 2023-11-30T21:52:42.617 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.614+0000 7f8f53fff700 1 -- 172.21.15.49:0/4290209754 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f5c014420 con 0x7f8f6c060280 2023-11-30T21:52:42.617 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.614+0000 7f8f71c4d700 1 -- 172.21.15.49:0/4290209754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6c045ad0 con 0x7f8f6c060280 2023-11-30T21:52:42.617 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.614+0000 7f8f53fff700 1 --2- 172.21.15.49:0/4290209754 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8f5402ff60 0x7f8f54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:42.617 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.614+0000 7f8f53fff700 1 -- 172.21.15.49:0/4290209754 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f5c0410f0 con 0x7f8f6c060280 2023-11-30T21:52:42.617 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.615+0000 7f8f6affd700 1 --2- 172.21.15.49:0/4290209754 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8f5402ff60 0x7f8f54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:42.618 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.615+0000 7f8f6affd700 1 --2- 172.21.15.49:0/4290209754 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8f5402ff60 0x7f8f54032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f8f600096a0 tx=0x7f8f60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:42.621 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.619+0000 7f8f53fff700 1 -- 172.21.15.49:0/4290209754 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f5c0095f0 con 0x7f8f6c060280 2023-11-30T21:52:42.839 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.838+0000 7f8f71c4d700 1 -- 172.21.15.49:0/4290209754 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f6c108110 con 0x7f8f6c060280 2023-11-30T21:52:42.840 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.838+0000 7f8f53fff700 1 -- 172.21.15.49:0/4290209754 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f5c01cb10 con 0x7f8f6c060280 2023-11-30T21:52:42.841 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:42.841 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:42.843 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.840+0000 7f8f71c4d700 1 -- 172.21.15.49:0/4290209754 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8f5402ff60 msgr2=0x7f8f54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:42.843 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.840+0000 7f8f71c4d700 1 --2- 172.21.15.49:0/4290209754 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8f5402ff60 0x7f8f54032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f8f600096a0 tx=0x7f8f60006b40).stop 2023-11-30T21:52:42.844 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.840+0000 7f8f71c4d700 1 -- 172.21.15.49:0/4290209754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f6c060280 msgr2=0x7f8f6c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:42.844 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.840+0000 7f8f71c4d700 1 --2- 172.21.15.49:0/4290209754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f6c060280 0x7f8f6c1066e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f8f5c013510 tx=0x7f8f5c00b040).stop 2023-11-30T21:52:42.845 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.840+0000 7f8f71c4d700 1 -- 172.21.15.49:0/4290209754 shutdown_connections 2023-11-30T21:52:42.845 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.840+0000 7f8f71c4d700 1 --2- 172.21.15.49:0/4290209754 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f8f5402ff60 0x7f8f54032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:42.845 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.840+0000 7f8f71c4d700 1 --2- 172.21.15.49:0/4290209754 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f6c060280 0x7f8f6c1066e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:42.845 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.840+0000 7f8f71c4d700 1 -- 172.21.15.49:0/4290209754 >> 172.21.15.49:0/4290209754 conn(0x7f8f6c0f0b70 msgr2=0x7f8f6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:42.845 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.840+0000 7f8f71c4d700 1 -- 172.21.15.49:0/4290209754 shutdown_connections 2023-11-30T21:52:42.846 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:42.840+0000 7f8f71c4d700 1 -- 172.21.15.49:0/4290209754 wait complete. 2023-11-30T21:52:42.846 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:43.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:41.541364+0000 mgr.smithi027.brrjcl (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:43.514 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:43 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:42.839813+0000 mon.smithi027 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.49:0/4290209754' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:44.263 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:44.263 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:45.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:45 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:43.541778+0000 mgr.smithi027.brrjcl (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:45.536 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.533+0000 7f0835fa0700 1 -- 172.21.15.49:0/3481781476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08300f78e0 msgr2=0x7f08300f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:45.536 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.533+0000 7f0835fa0700 1 --2- 172.21.15.49:0/3481781476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08300f78e0 0x7f08300f7d00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f0824009fb0 tx=0x7f0824009b30).stop 2023-11-30T21:52:45.536 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.533+0000 7f0835fa0700 1 -- 172.21.15.49:0/3481781476 shutdown_connections 2023-11-30T21:52:45.536 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.533+0000 7f0835fa0700 1 --2- 172.21.15.49:0/3481781476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08300f78e0 0x7f08300f7d00 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:45.536 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.533+0000 7f0835fa0700 1 -- 172.21.15.49:0/3481781476 >> 172.21.15.49:0/3481781476 conn(0x7f08300f0120 msgr2=0x7f08300f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:45.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.534+0000 7f0835fa0700 1 -- 172.21.15.49:0/3481781476 shutdown_connections 2023-11-30T21:52:45.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.534+0000 7f0835fa0700 1 -- 172.21.15.49:0/3481781476 wait complete. 2023-11-30T21:52:45.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.534+0000 7f0835fa0700 1 Processor -- start 2023-11-30T21:52:45.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.534+0000 7f0835fa0700 1 -- start start 2023-11-30T21:52:45.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.535+0000 7f0835fa0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08300f78e0 0x7f08300f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:45.537 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.535+0000 7f0835fa0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08300f5630 con 0x7f08300f78e0 2023-11-30T21:52:45.538 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.535+0000 7f0834f9e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08300f78e0 0x7f08300f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:45.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.535+0000 7f0834f9e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08300f78e0 0x7f08300f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:41500/0 (socket says 172.21.15.49:41500) 2023-11-30T21:52:45.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.535+0000 7f0834f9e700 1 -- 172.21.15.49:0/1447446516 learned_addr learned my addr 172.21.15.49:0/1447446516 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:45.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.535+0000 7f0834f9e700 1 -- 172.21.15.49:0/1447446516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0824005040 con 0x7f08300f78e0 2023-11-30T21:52:45.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.536+0000 7f0834f9e700 1 --2- 172.21.15.49:0/1447446516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08300f78e0 0x7f08300f71b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f0824013510 tx=0x7f082400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:45.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.536+0000 7f082d7fa700 1 -- 172.21.15.49:0/1447446516 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0824009320 con 0x7f08300f78e0 2023-11-30T21:52:45.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.536+0000 7f082d7fa700 1 -- 172.21.15.49:0/1447446516 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0824013f00 con 0x7f08300f78e0 2023-11-30T21:52:45.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.536+0000 7f0835fa0700 1 -- 172.21.15.49:0/1447446516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08300f57d0 con 0x7f08300f78e0 2023-11-30T21:52:45.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.536+0000 7f082d7fa700 1 -- 172.21.15.49:0/1447446516 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0824014dc0 con 0x7f08300f78e0 2023-11-30T21:52:45.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.536+0000 7f0835fa0700 1 -- 172.21.15.49:0/1447446516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08300f5c60 con 0x7f08300f78e0 2023-11-30T21:52:45.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.537+0000 7f082d7fa700 1 -- 172.21.15.49:0/1447446516 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f08240095f0 con 0x7f08300f78e0 2023-11-30T21:52:45.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.537+0000 7f0835fa0700 1 -- 172.21.15.49:0/1447446516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0830059100 con 0x7f08300f78e0 2023-11-30T21:52:45.539 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.537+0000 7f082d7fa700 1 --2- 172.21.15.49:0/1447446516 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f082002ff60 0x7f0820032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:45.540 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.537+0000 7f082d7fa700 1 -- 172.21.15.49:0/1447446516 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0824041040 con 0x7f08300f78e0 2023-11-30T21:52:45.540 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.537+0000 7f082ffff700 1 --2- 172.21.15.49:0/1447446516 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f082002ff60 0x7f0820032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:45.540 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.537+0000 7f082ffff700 1 --2- 172.21.15.49:0/1447446516 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f082002ff60 0x7f0820032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f081c0096a0 tx=0x7f081c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:45.543 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.542+0000 7f082d7fa700 1 -- 172.21.15.49:0/1447446516 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0824014420 con 0x7f08300f78e0 2023-11-30T21:52:45.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.761+0000 7f0835fa0700 1 -- 172.21.15.49:0/1447446516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08300fbce0 con 0x7f08300f78e0 2023-11-30T21:52:45.763 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.761+0000 7f082d7fa700 1 -- 172.21.15.49:0/1447446516 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0824010230 con 0x7f08300f78e0 2023-11-30T21:52:45.763 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:45.763 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:45.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.763+0000 7f0835fa0700 1 -- 172.21.15.49:0/1447446516 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f082002ff60 msgr2=0x7f0820032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:45.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.763+0000 7f0835fa0700 1 --2- 172.21.15.49:0/1447446516 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f082002ff60 0x7f0820032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f081c0096a0 tx=0x7f081c006b40).stop 2023-11-30T21:52:45.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.763+0000 7f0835fa0700 1 -- 172.21.15.49:0/1447446516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08300f78e0 msgr2=0x7f08300f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:45.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.763+0000 7f0835fa0700 1 --2- 172.21.15.49:0/1447446516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08300f78e0 0x7f08300f71b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f0824013510 tx=0x7f082400b040).stop 2023-11-30T21:52:45.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.763+0000 7f0835fa0700 1 -- 172.21.15.49:0/1447446516 shutdown_connections 2023-11-30T21:52:45.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.763+0000 7f0835fa0700 1 --2- 172.21.15.49:0/1447446516 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f082002ff60 0x7f0820032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:45.765 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.763+0000 7f0835fa0700 1 --2- 172.21.15.49:0/1447446516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08300f78e0 0x7f08300f71b0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:45.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.763+0000 7f0835fa0700 1 -- 172.21.15.49:0/1447446516 >> 172.21.15.49:0/1447446516 conn(0x7f08300f0120 msgr2=0x7f08300f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:45.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.763+0000 7f0835fa0700 1 -- 172.21.15.49:0/1447446516 shutdown_connections 2023-11-30T21:52:45.766 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:45.763+0000 7f0835fa0700 1 -- 172.21.15.49:0/1447446516 wait complete. 2023-11-30T21:52:45.766 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:46.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:46 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:45.762859+0000 mon.smithi027 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.49:0/1447446516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:47.174 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:47.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:47.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:47 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:45.542015+0000 mgr.smithi027.brrjcl (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:48.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.422+0000 7f6e362e9700 1 -- 172.21.15.49:0/957644703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e300f5410 msgr2=0x7f6e300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:48.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.422+0000 7f6e362e9700 1 --2- 172.21.15.49:0/957644703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e300f5410 0x7f6e300f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f6e20009fb0 tx=0x7f6e20009b30).stop 2023-11-30T21:52:48.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.422+0000 7f6e362e9700 1 -- 172.21.15.49:0/957644703 shutdown_connections 2023-11-30T21:52:48.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.422+0000 7f6e362e9700 1 --2- 172.21.15.49:0/957644703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e300f5410 0x7f6e300f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:48.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.422+0000 7f6e362e9700 1 -- 172.21.15.49:0/957644703 >> 172.21.15.49:0/957644703 conn(0x7f6e300f0b70 msgr2=0x7f6e300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:48.425 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.422+0000 7f6e362e9700 1 -- 172.21.15.49:0/957644703 shutdown_connections 2023-11-30T21:52:48.426 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.422+0000 7f6e362e9700 1 -- 172.21.15.49:0/957644703 wait complete. 2023-11-30T21:52:48.427 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.423+0000 7f6e362e9700 1 Processor -- start 2023-11-30T21:52:48.427 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.423+0000 7f6e362e9700 1 -- start start 2023-11-30T21:52:48.431 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.423+0000 7f6e362e9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e300f5410 0x7f6e30105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:48.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.423+0000 7f6e362e9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e30105b20 con 0x7f6e300f5410 2023-11-30T21:52:48.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.424+0000 7f6e2ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e300f5410 0x7f6e30105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:48.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.424+0000 7f6e2ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e300f5410 0x7f6e30105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:41528/0 (socket says 172.21.15.49:41528) 2023-11-30T21:52:48.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.424+0000 7f6e2ffff700 1 -- 172.21.15.49:0/3871115626 learned_addr learned my addr 172.21.15.49:0/3871115626 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:48.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.424+0000 7f6e2ffff700 1 -- 172.21.15.49:0/3871115626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e20005040 con 0x7f6e300f5410 2023-11-30T21:52:48.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.424+0000 7f6e2ffff700 1 --2- 172.21.15.49:0/3871115626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e300f5410 0x7f6e30105630 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f6e20013510 tx=0x7f6e2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:48.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.424+0000 7f6e2cff9700 1 -- 172.21.15.49:0/3871115626 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e20009320 con 0x7f6e300f5410 2023-11-30T21:52:48.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.424+0000 7f6e2cff9700 1 -- 172.21.15.49:0/3871115626 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e20013f00 con 0x7f6e300f5410 2023-11-30T21:52:48.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.424+0000 7f6e362e9700 1 -- 172.21.15.49:0/3871115626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e30105d20 con 0x7f6e300f5410 2023-11-30T21:52:48.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.425+0000 7f6e2cff9700 1 -- 172.21.15.49:0/3871115626 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e20014d90 con 0x7f6e300f5410 2023-11-30T21:52:48.432 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.425+0000 7f6e362e9700 1 -- 172.21.15.49:0/3871115626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e301061b0 con 0x7f6e300f5410 2023-11-30T21:52:48.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.425+0000 7f6e2cff9700 1 -- 172.21.15.49:0/3871115626 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e20014420 con 0x7f6e300f5410 2023-11-30T21:52:48.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.425+0000 7f6e362e9700 1 -- 172.21.15.49:0/3871115626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e30045ad0 con 0x7f6e300f5410 2023-11-30T21:52:48.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.425+0000 7f6e2cff9700 1 --2- 172.21.15.49:0/3871115626 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6e1802ff60 0x7f6e18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:48.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.426+0000 7f6e2cff9700 1 -- 172.21.15.49:0/3871115626 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e20041000 con 0x7f6e300f5410 2023-11-30T21:52:48.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.426+0000 7f6e2f7fe700 1 --2- 172.21.15.49:0/3871115626 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6e1802ff60 0x7f6e18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:48.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.426+0000 7f6e2f7fe700 1 --2- 172.21.15.49:0/3871115626 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6e1802ff60 0x7f6e18032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f6e240096a0 tx=0x7f6e24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:48.433 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.431+0000 7f6e2cff9700 1 -- 172.21.15.49:0/3871115626 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e200095f0 con 0x7f6e300f5410 2023-11-30T21:52:48.654 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.653+0000 7f6e362e9700 1 -- 172.21.15.49:0/3871115626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e300fae60 con 0x7f6e300f5410 2023-11-30T21:52:48.655 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.654+0000 7f6e2cff9700 1 -- 172.21.15.49:0/3871115626 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e2001db70 con 0x7f6e300f5410 2023-11-30T21:52:48.655 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:48.655 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:48.657 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.655+0000 7f6e362e9700 1 -- 172.21.15.49:0/3871115626 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6e1802ff60 msgr2=0x7f6e18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:48.657 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.655+0000 7f6e362e9700 1 --2- 172.21.15.49:0/3871115626 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6e1802ff60 0x7f6e18032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f6e240096a0 tx=0x7f6e24006b40).stop 2023-11-30T21:52:48.657 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.655+0000 7f6e362e9700 1 -- 172.21.15.49:0/3871115626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e300f5410 msgr2=0x7f6e30105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:48.657 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.655+0000 7f6e362e9700 1 --2- 172.21.15.49:0/3871115626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e300f5410 0x7f6e30105630 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f6e20013510 tx=0x7f6e2000b040).stop 2023-11-30T21:52:48.657 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.655+0000 7f6e362e9700 1 -- 172.21.15.49:0/3871115626 shutdown_connections 2023-11-30T21:52:48.657 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.655+0000 7f6e362e9700 1 --2- 172.21.15.49:0/3871115626 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f6e1802ff60 0x7f6e18032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:48.657 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.655+0000 7f6e362e9700 1 --2- 172.21.15.49:0/3871115626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e300f5410 0x7f6e30105630 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:48.658 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.655+0000 7f6e362e9700 1 -- 172.21.15.49:0/3871115626 >> 172.21.15.49:0/3871115626 conn(0x7f6e300f0b70 msgr2=0x7f6e300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:48.658 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.655+0000 7f6e362e9700 1 -- 172.21.15.49:0/3871115626 shutdown_connections 2023-11-30T21:52:48.658 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:48.655+0000 7f6e362e9700 1 -- 172.21.15.49:0/3871115626 wait complete. 2023-11-30T21:52:48.658 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:49.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:49 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:47.542365+0000 mgr.smithi027.brrjcl (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:49.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:49 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:48.655148+0000 mon.smithi027 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.49:0/3871115626' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:50.076 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:50.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:51.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.313+0000 7f5d267db700 1 -- 172.21.15.49:0/403851075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d200f5410 msgr2=0x7f5d200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:51.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.313+0000 7f5d267db700 1 --2- 172.21.15.49:0/403851075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d200f5410 0x7f5d200f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f5d10009fb0 tx=0x7f5d10009b30).stop 2023-11-30T21:52:51.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.314+0000 7f5d267db700 1 -- 172.21.15.49:0/403851075 shutdown_connections 2023-11-30T21:52:51.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.314+0000 7f5d267db700 1 --2- 172.21.15.49:0/403851075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d200f5410 0x7f5d200f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:51.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.314+0000 7f5d267db700 1 -- 172.21.15.49:0/403851075 >> 172.21.15.49:0/403851075 conn(0x7f5d200f0b70 msgr2=0x7f5d200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:51.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.314+0000 7f5d267db700 1 -- 172.21.15.49:0/403851075 shutdown_connections 2023-11-30T21:52:51.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.314+0000 7f5d267db700 1 -- 172.21.15.49:0/403851075 wait complete. 2023-11-30T21:52:51.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.314+0000 7f5d267db700 1 Processor -- start 2023-11-30T21:52:51.316 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.314+0000 7f5d267db700 1 -- start start 2023-11-30T21:52:51.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.315+0000 7f5d267db700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d200f5410 0x7f5d20109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:51.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.315+0000 7f5d267db700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d20060220 con 0x7f5d200f5410 2023-11-30T21:52:51.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.315+0000 7f5d1ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d200f5410 0x7f5d20109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:51.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.315+0000 7f5d1ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d200f5410 0x7f5d20109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:45102/0 (socket says 172.21.15.49:45102) 2023-11-30T21:52:51.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.315+0000 7f5d1ffff700 1 -- 172.21.15.49:0/2384895695 learned_addr learned my addr 172.21.15.49:0/2384895695 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:51.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.315+0000 7f5d1ffff700 1 -- 172.21.15.49:0/2384895695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d10005040 con 0x7f5d200f5410 2023-11-30T21:52:51.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.315+0000 7f5d1ffff700 1 --2- 172.21.15.49:0/2384895695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d200f5410 0x7f5d20109450 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f5d10013510 tx=0x7f5d1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:51.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.316+0000 7f5d1cff9700 1 -- 172.21.15.49:0/2384895695 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d10009320 con 0x7f5d200f5410 2023-11-30T21:52:51.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.316+0000 7f5d267db700 1 -- 172.21.15.49:0/2384895695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d20060420 con 0x7f5d200f5410 2023-11-30T21:52:51.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.316+0000 7f5d267db700 1 -- 172.21.15.49:0/2384895695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d20109b00 con 0x7f5d200f5410 2023-11-30T21:52:51.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.316+0000 7f5d1cff9700 1 -- 172.21.15.49:0/2384895695 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d10013f00 con 0x7f5d200f5410 2023-11-30T21:52:51.318 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.316+0000 7f5d1cff9700 1 -- 172.21.15.49:0/2384895695 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d10014d60 con 0x7f5d200f5410 2023-11-30T21:52:51.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.317+0000 7f5d267db700 1 -- 172.21.15.49:0/2384895695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d00004fa0 con 0x7f5d200f5410 2023-11-30T21:52:51.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.317+0000 7f5d1cff9700 1 -- 172.21.15.49:0/2384895695 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d100095f0 con 0x7f5d200f5410 2023-11-30T21:52:51.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.317+0000 7f5d1cff9700 1 --2- 172.21.15.49:0/2384895695 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5d0802ff60 0x7f5d08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:51.319 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.317+0000 7f5d1cff9700 1 -- 172.21.15.49:0/2384895695 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d10040f50 con 0x7f5d200f5410 2023-11-30T21:52:51.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.317+0000 7f5d1f7fe700 1 --2- 172.21.15.49:0/2384895695 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5d0802ff60 0x7f5d08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:51.320 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.317+0000 7f5d1f7fe700 1 --2- 172.21.15.49:0/2384895695 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5d0802ff60 0x7f5d08032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f5d140096a0 tx=0x7f5d14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:51.323 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.322+0000 7f5d1cff9700 1 -- 172.21.15.49:0/2384895695 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d1006d2b0 con 0x7f5d200f5410 2023-11-30T21:52:51.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:51 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:49.542776+0000 mgr.smithi027.brrjcl (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:51.544 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.542+0000 7f5d267db700 1 -- 172.21.15.49:0/2384895695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d00005670 con 0x7f5d200f5410 2023-11-30T21:52:51.544 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.543+0000 7f5d1cff9700 1 -- 172.21.15.49:0/2384895695 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d10014420 con 0x7f5d200f5410 2023-11-30T21:52:51.545 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:51.545 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:51.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.544+0000 7f5d267db700 1 -- 172.21.15.49:0/2384895695 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5d0802ff60 msgr2=0x7f5d08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:51.546 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.544+0000 7f5d267db700 1 --2- 172.21.15.49:0/2384895695 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5d0802ff60 0x7f5d08032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f5d140096a0 tx=0x7f5d14006b40).stop 2023-11-30T21:52:51.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.544+0000 7f5d267db700 1 -- 172.21.15.49:0/2384895695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d200f5410 msgr2=0x7f5d20109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:51.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.544+0000 7f5d267db700 1 --2- 172.21.15.49:0/2384895695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d200f5410 0x7f5d20109450 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f5d10013510 tx=0x7f5d1000b040).stop 2023-11-30T21:52:51.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.545+0000 7f5d267db700 1 -- 172.21.15.49:0/2384895695 shutdown_connections 2023-11-30T21:52:51.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.545+0000 7f5d267db700 1 --2- 172.21.15.49:0/2384895695 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5d0802ff60 0x7f5d08032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:51.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.545+0000 7f5d267db700 1 --2- 172.21.15.49:0/2384895695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d200f5410 0x7f5d20109450 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:51.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.545+0000 7f5d267db700 1 -- 172.21.15.49:0/2384895695 >> 172.21.15.49:0/2384895695 conn(0x7f5d200f0b70 msgr2=0x7f5d200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:51.547 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.545+0000 7f5d267db700 1 -- 172.21.15.49:0/2384895695 shutdown_connections 2023-11-30T21:52:51.548 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:51.545+0000 7f5d267db700 1 -- 172.21.15.49:0/2384895695 wait complete. 2023-11-30T21:52:51.548 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:52.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:52 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:51.544630+0000 mon.smithi027 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.49:0/2384895695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:52.970 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:52.970 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:53.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:53 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:51.543056+0000 mgr.smithi027.brrjcl (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:54.214 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.211+0000 7f5e2359e700 1 -- 172.21.15.49:0/3949590640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e240f51a0 msgr2=0x7f5e240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:54.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.211+0000 7f5e2359e700 1 --2- 172.21.15.49:0/3949590640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e240f51a0 0x7f5e240f55c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f5e14009fb0 tx=0x7f5e14009b30).stop 2023-11-30T21:52:54.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.211+0000 7f5e2359e700 1 -- 172.21.15.49:0/3949590640 shutdown_connections 2023-11-30T21:52:54.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.211+0000 7f5e2359e700 1 --2- 172.21.15.49:0/3949590640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e240f51a0 0x7f5e240f55c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:54.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.211+0000 7f5e2359e700 1 -- 172.21.15.49:0/3949590640 >> 172.21.15.49:0/3949590640 conn(0x7f5e240f09a0 msgr2=0x7f5e240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:54.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.211+0000 7f5e2359e700 1 -- 172.21.15.49:0/3949590640 shutdown_connections 2023-11-30T21:52:54.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.211+0000 7f5e2359e700 1 -- 172.21.15.49:0/3949590640 wait complete. 2023-11-30T21:52:54.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.212+0000 7f5e2359e700 1 Processor -- start 2023-11-30T21:52:54.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.212+0000 7f5e2359e700 1 -- start start 2023-11-30T21:52:54.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.212+0000 7f5e2359e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e240f51a0 0x7f5e241091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:54.215 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.212+0000 7f5e2359e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e24060220 con 0x7f5e240f51a0 2023-11-30T21:52:54.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.212+0000 7f5e2259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e240f51a0 0x7f5e241091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:54.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.212+0000 7f5e2259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e240f51a0 0x7f5e241091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:45120/0 (socket says 172.21.15.49:45120) 2023-11-30T21:52:54.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.212+0000 7f5e2259c700 1 -- 172.21.15.49:0/3836372669 learned_addr learned my addr 172.21.15.49:0/3836372669 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:54.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.212+0000 7f5e2259c700 1 -- 172.21.15.49:0/3836372669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e14005040 con 0x7f5e240f51a0 2023-11-30T21:52:54.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.212+0000 7f5e2259c700 1 --2- 172.21.15.49:0/3836372669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e240f51a0 0x7f5e241091d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f5e14013510 tx=0x7f5e1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:54.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.213+0000 7f5e12ffd700 1 -- 172.21.15.49:0/3836372669 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e14009320 con 0x7f5e240f51a0 2023-11-30T21:52:54.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.213+0000 7f5e12ffd700 1 -- 172.21.15.49:0/3836372669 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e14013f00 con 0x7f5e240f51a0 2023-11-30T21:52:54.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.213+0000 7f5e2359e700 1 -- 172.21.15.49:0/3836372669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e24060420 con 0x7f5e240f51a0 2023-11-30T21:52:54.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.213+0000 7f5e12ffd700 1 -- 172.21.15.49:0/3836372669 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e14014dc0 con 0x7f5e240f51a0 2023-11-30T21:52:54.216 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.213+0000 7f5e2359e700 1 -- 172.21.15.49:0/3836372669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e24109880 con 0x7f5e240f51a0 2023-11-30T21:52:54.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.214+0000 7f5e12ffd700 1 -- 172.21.15.49:0/3836372669 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e140095f0 con 0x7f5e240f51a0 2023-11-30T21:52:54.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.214+0000 7f5e2359e700 1 -- 172.21.15.49:0/3836372669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e24059100 con 0x7f5e240f51a0 2023-11-30T21:52:54.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.214+0000 7f5e12ffd700 1 --2- 172.21.15.49:0/3836372669 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5e0c02ffb0 0x7f5e0c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:54.217 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.214+0000 7f5e12ffd700 1 -- 172.21.15.49:0/3836372669 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e14040fe0 con 0x7f5e240f51a0 2023-11-30T21:52:54.218 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.214+0000 7f5e21d9b700 1 --2- 172.21.15.49:0/3836372669 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5e0c02ffb0 0x7f5e0c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:54.218 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.214+0000 7f5e21d9b700 1 --2- 172.21.15.49:0/3836372669 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5e0c02ffb0 0x7f5e0c032470 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f5e180096a0 tx=0x7f5e18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:54.220 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.219+0000 7f5e12ffd700 1 -- 172.21.15.49:0/3836372669 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e1406d2b0 con 0x7f5e240f51a0 2023-11-30T21:52:54.437 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.436+0000 7f5e2359e700 1 -- 172.21.15.49:0/3836372669 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e24045ad0 con 0x7f5e240f51a0 2023-11-30T21:52:54.438 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.436+0000 7f5e12ffd700 1 -- 172.21.15.49:0/3836372669 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e14014420 con 0x7f5e240f51a0 2023-11-30T21:52:54.438 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:54.438 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:54.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.439+0000 7f5e2359e700 1 -- 172.21.15.49:0/3836372669 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5e0c02ffb0 msgr2=0x7f5e0c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:54.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.439+0000 7f5e2359e700 1 --2- 172.21.15.49:0/3836372669 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5e0c02ffb0 0x7f5e0c032470 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f5e180096a0 tx=0x7f5e18006b40).stop 2023-11-30T21:52:54.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.439+0000 7f5e2359e700 1 -- 172.21.15.49:0/3836372669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e240f51a0 msgr2=0x7f5e241091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:54.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.439+0000 7f5e2359e700 1 --2- 172.21.15.49:0/3836372669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e240f51a0 0x7f5e241091d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f5e14013510 tx=0x7f5e1400b040).stop 2023-11-30T21:52:54.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.439+0000 7f5e2359e700 1 -- 172.21.15.49:0/3836372669 shutdown_connections 2023-11-30T21:52:54.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.439+0000 7f5e2359e700 1 --2- 172.21.15.49:0/3836372669 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f5e0c02ffb0 0x7f5e0c032470 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:54.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.439+0000 7f5e2359e700 1 --2- 172.21.15.49:0/3836372669 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e240f51a0 0x7f5e241091d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:54.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.439+0000 7f5e2359e700 1 -- 172.21.15.49:0/3836372669 >> 172.21.15.49:0/3836372669 conn(0x7f5e240f09a0 msgr2=0x7f5e240f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:54.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.439+0000 7f5e2359e700 1 -- 172.21.15.49:0/3836372669 shutdown_connections 2023-11-30T21:52:54.440 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:54.439+0000 7f5e2359e700 1 -- 172.21.15.49:0/3836372669 wait complete. 2023-11-30T21:52:54.441 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:55.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:53.543394+0000 mgr.smithi027.brrjcl (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:55.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:55 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:54.438005+0000 mon.smithi027 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.49:0/3836372669' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:55.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:55.883 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:57.204 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.202+0000 7fb328f9c700 1 -- 172.21.15.49:0/822687910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3240f5410 msgr2=0x7fb3240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:57.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.202+0000 7fb328f9c700 1 --2- 172.21.15.49:0/822687910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3240f5410 0x7fb3240f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fb314009fb0 tx=0x7fb314009b30).stop 2023-11-30T21:52:57.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.202+0000 7fb328f9c700 1 -- 172.21.15.49:0/822687910 shutdown_connections 2023-11-30T21:52:57.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.202+0000 7fb328f9c700 1 --2- 172.21.15.49:0/822687910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3240f5410 0x7fb3240f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:57.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.202+0000 7fb328f9c700 1 -- 172.21.15.49:0/822687910 >> 172.21.15.49:0/822687910 conn(0x7fb3240f0b70 msgr2=0x7fb3240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:57.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.202+0000 7fb328f9c700 1 -- 172.21.15.49:0/822687910 shutdown_connections 2023-11-30T21:52:57.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.202+0000 7fb328f9c700 1 -- 172.21.15.49:0/822687910 wait complete. 2023-11-30T21:52:57.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.203+0000 7fb328f9c700 1 Processor -- start 2023-11-30T21:52:57.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.203+0000 7fb328f9c700 1 -- start start 2023-11-30T21:52:57.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.203+0000 7fb328f9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3240f5410 0x7fb324105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:57.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.203+0000 7fb328f9c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb324105b20 con 0x7fb3240f5410 2023-11-30T21:52:57.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.203+0000 7fb32259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3240f5410 0x7fb324105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:57.205 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.203+0000 7fb32259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3240f5410 0x7fb324105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:45138/0 (socket says 172.21.15.49:45138) 2023-11-30T21:52:57.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.203+0000 7fb32259c700 1 -- 172.21.15.49:0/1697151720 learned_addr learned my addr 172.21.15.49:0/1697151720 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:52:57.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.204+0000 7fb32259c700 1 -- 172.21.15.49:0/1697151720 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb314005040 con 0x7fb3240f5410 2023-11-30T21:52:57.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.204+0000 7fb32259c700 1 --2- 172.21.15.49:0/1697151720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3240f5410 0x7fb324105630 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fb314013510 tx=0x7fb31400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:57.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.204+0000 7fb31affd700 1 -- 172.21.15.49:0/1697151720 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb314009320 con 0x7fb3240f5410 2023-11-30T21:52:57.206 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.204+0000 7fb328f9c700 1 -- 172.21.15.49:0/1697151720 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb324105d20 con 0x7fb3240f5410 2023-11-30T21:52:57.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.204+0000 7fb31affd700 1 -- 172.21.15.49:0/1697151720 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb314013f00 con 0x7fb3240f5410 2023-11-30T21:52:57.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.204+0000 7fb31affd700 1 -- 172.21.15.49:0/1697151720 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb314014d90 con 0x7fb3240f5410 2023-11-30T21:52:57.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.204+0000 7fb328f9c700 1 -- 172.21.15.49:0/1697151720 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3241061b0 con 0x7fb3240f5410 2023-11-30T21:52:57.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.205+0000 7fb31affd700 1 -- 172.21.15.49:0/1697151720 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3140095f0 con 0x7fb3240f5410 2023-11-30T21:52:57.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.205+0000 7fb328f9c700 1 -- 172.21.15.49:0/1697151720 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb324045ad0 con 0x7fb3240f5410 2023-11-30T21:52:57.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.205+0000 7fb31affd700 1 --2- 172.21.15.49:0/1697151720 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb31002ff60 0x7fb310032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:52:57.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.205+0000 7fb31affd700 1 -- 172.21.15.49:0/1697151720 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb314040ff0 con 0x7fb3240f5410 2023-11-30T21:52:57.207 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.205+0000 7fb321d9b700 1 --2- 172.21.15.49:0/1697151720 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb31002ff60 0x7fb310032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:52:57.208 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.206+0000 7fb321d9b700 1 --2- 172.21.15.49:0/1697151720 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb31002ff60 0x7fb310032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fb30c0096a0 tx=0x7fb30c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:52:57.210 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.210+0000 7fb31affd700 1 -- 172.21.15.49:0/1697151720 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb314014420 con 0x7fb3240f5410 2023-11-30T21:52:57.426 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.425+0000 7fb328f9c700 1 -- 172.21.15.49:0/1697151720 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3240fae60 con 0x7fb3240f5410 2023-11-30T21:52:57.427 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.426+0000 7fb31affd700 1 -- 172.21.15.49:0/1697151720 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb314010230 con 0x7fb3240f5410 2023-11-30T21:52:57.427 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:52:57.427 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:52:57.428 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.427+0000 7fb328f9c700 1 -- 172.21.15.49:0/1697151720 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb31002ff60 msgr2=0x7fb310032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:57.428 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.427+0000 7fb328f9c700 1 --2- 172.21.15.49:0/1697151720 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb31002ff60 0x7fb310032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fb30c0096a0 tx=0x7fb30c006b40).stop 2023-11-30T21:52:57.428 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.427+0000 7fb328f9c700 1 -- 172.21.15.49:0/1697151720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3240f5410 msgr2=0x7fb324105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:52:57.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.427+0000 7fb328f9c700 1 --2- 172.21.15.49:0/1697151720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3240f5410 0x7fb324105630 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fb314013510 tx=0x7fb31400b040).stop 2023-11-30T21:52:57.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.427+0000 7fb328f9c700 1 -- 172.21.15.49:0/1697151720 shutdown_connections 2023-11-30T21:52:57.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.427+0000 7fb328f9c700 1 --2- 172.21.15.49:0/1697151720 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fb31002ff60 0x7fb310032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:57.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.427+0000 7fb328f9c700 1 --2- 172.21.15.49:0/1697151720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3240f5410 0x7fb324105630 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:52:57.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.427+0000 7fb328f9c700 1 -- 172.21.15.49:0/1697151720 >> 172.21.15.49:0/1697151720 conn(0x7fb3240f0b70 msgr2=0x7fb3240f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:52:57.429 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.427+0000 7fb328f9c700 1 -- 172.21.15.49:0/1697151720 shutdown_connections 2023-11-30T21:52:57.430 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:52:57.427+0000 7fb328f9c700 1 -- 172.21.15.49:0/1697151720 wait complete. 2023-11-30T21:52:57.430 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:52:57.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:57 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:55.543784+0000 mgr.smithi027.brrjcl (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:52:58.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:58 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:52:57.426418+0000 mon.smithi027 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.49:0/1697151720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:52:58.807 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:52:58.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:52:59.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:52:59 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:57.544013+0000 mgr.smithi027.brrjcl (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:00.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.076+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640744093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b80f62a0 msgr2=0x7f98b80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:00.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.076+0000 7f98bcf24700 1 --2- 172.21.15.49:0/1640744093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b80f62a0 0x7f98b80f66c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f98a8009fb0 tx=0x7f98a8009b30).stop 2023-11-30T21:53:00.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.076+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640744093 shutdown_connections 2023-11-30T21:53:00.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.076+0000 7f98bcf24700 1 --2- 172.21.15.49:0/1640744093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b80f62a0 0x7f98b80f66c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:00.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.076+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640744093 >> 172.21.15.49:0/1640744093 conn(0x7f98b80f1aa0 msgr2=0x7f98b80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:00.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.076+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640744093 shutdown_connections 2023-11-30T21:53:00.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.076+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640744093 wait complete. 2023-11-30T21:53:00.078 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.076+0000 7f98bcf24700 1 Processor -- start 2023-11-30T21:53:00.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.077+0000 7f98bcf24700 1 -- start start 2023-11-30T21:53:00.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.077+0000 7f98bcf24700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b80f62a0 0x7f98b81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:00.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.077+0000 7f98bcf24700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b81069a0 con 0x7f98b80f62a0 2023-11-30T21:53:00.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.077+0000 7f98b77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b80f62a0 0x7f98b81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:00.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.077+0000 7f98b77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b80f62a0 0x7f98b81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:55900/0 (socket says 172.21.15.49:55900) 2023-11-30T21:53:00.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.077+0000 7f98b77fe700 1 -- 172.21.15.49:0/1640777579 learned_addr learned my addr 172.21.15.49:0/1640777579 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:53:00.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.077+0000 7f98b77fe700 1 -- 172.21.15.49:0/1640777579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a8005040 con 0x7f98b80f62a0 2023-11-30T21:53:00.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.077+0000 7f98b77fe700 1 --2- 172.21.15.49:0/1640777579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b80f62a0 0x7f98b81064b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f98a8013510 tx=0x7f98a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:00.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.078+0000 7f989ffff700 1 -- 172.21.15.49:0/1640777579 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98a8009320 con 0x7f98b80f62a0 2023-11-30T21:53:00.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.078+0000 7f989ffff700 1 -- 172.21.15.49:0/1640777579 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98a8013f00 con 0x7f98b80f62a0 2023-11-30T21:53:00.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.078+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640777579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b8106ba0 con 0x7f98b80f62a0 2023-11-30T21:53:00.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.078+0000 7f989ffff700 1 -- 172.21.15.49:0/1640777579 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98a8014dc0 con 0x7f98b80f62a0 2023-11-30T21:53:00.080 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.078+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640777579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b8107030 con 0x7f98b80f62a0 2023-11-30T21:53:00.081 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.078+0000 7f989ffff700 1 -- 172.21.15.49:0/1640777579 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98a80095f0 con 0x7f98b80f62a0 2023-11-30T21:53:00.081 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.078+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640777579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b8045ad0 con 0x7f98b80f62a0 2023-11-30T21:53:00.081 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.079+0000 7f989ffff700 1 --2- 172.21.15.49:0/1640777579 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f98a002ff60 0x7f98a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:00.081 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.079+0000 7f989ffff700 1 -- 172.21.15.49:0/1640777579 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98a8041040 con 0x7f98b80f62a0 2023-11-30T21:53:00.081 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.079+0000 7f98b6ffd700 1 --2- 172.21.15.49:0/1640777579 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f98a002ff60 0x7f98a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:00.081 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.079+0000 7f98b6ffd700 1 --2- 172.21.15.49:0/1640777579 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f98a002ff60 0x7f98a0032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f98ac0096a0 tx=0x7f98ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:00.084 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.084+0000 7f989ffff700 1 -- 172.21.15.49:0/1640777579 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98a8014420 con 0x7f98b80f62a0 2023-11-30T21:53:00.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.302+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640777579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98b8107ee0 con 0x7f98b80f62a0 2023-11-30T21:53:00.303 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.303+0000 7f989ffff700 1 -- 172.21.15.49:0/1640777579 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f98a8010230 con 0x7f98b80f62a0 2023-11-30T21:53:00.303 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:53:00.303 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:53:00.305 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.304+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640777579 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f98a002ff60 msgr2=0x7f98a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:00.305 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.304+0000 7f98bcf24700 1 --2- 172.21.15.49:0/1640777579 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f98a002ff60 0x7f98a0032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f98ac0096a0 tx=0x7f98ac006b40).stop 2023-11-30T21:53:00.305 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.304+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640777579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b80f62a0 msgr2=0x7f98b81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:00.305 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.304+0000 7f98bcf24700 1 --2- 172.21.15.49:0/1640777579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b80f62a0 0x7f98b81064b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f98a8013510 tx=0x7f98a800b040).stop 2023-11-30T21:53:00.305 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.304+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640777579 shutdown_connections 2023-11-30T21:53:00.305 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.304+0000 7f98bcf24700 1 --2- 172.21.15.49:0/1640777579 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f98a002ff60 0x7f98a0032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:00.305 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.304+0000 7f98bcf24700 1 --2- 172.21.15.49:0/1640777579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b80f62a0 0x7f98b81064b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:00.306 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.304+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640777579 >> 172.21.15.49:0/1640777579 conn(0x7f98b80f1aa0 msgr2=0x7f98b80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:00.306 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.304+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640777579 shutdown_connections 2023-11-30T21:53:00.306 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:00.304+0000 7f98bcf24700 1 -- 172.21.15.49:0/1640777579 wait complete. 2023-11-30T21:53:00.306 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:53:01.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:01 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:52:59.544376+0000 mgr.smithi027.brrjcl (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:01.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:01 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:53:00.303043+0000 mon.smithi027 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.49:0/1640777579' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:53:01.731 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:53:01.732 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:53:02.954 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.953+0000 7f791e6d1700 1 -- 172.21.15.49:0/3687238817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79180601e0 msgr2=0x7f79180f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:02.955 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.953+0000 7f791e6d1700 1 --2- 172.21.15.49:0/3687238817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79180601e0 0x7f79180f8f20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f7914009960 tx=0x7f7914004d10).stop 2023-11-30T21:53:02.955 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.953+0000 7f791e6d1700 1 -- 172.21.15.49:0/3687238817 shutdown_connections 2023-11-30T21:53:02.955 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.953+0000 7f791e6d1700 1 --2- 172.21.15.49:0/3687238817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79180601e0 0x7f79180f8f20 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:02.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.953+0000 7f791e6d1700 1 -- 172.21.15.49:0/3687238817 >> 172.21.15.49:0/3687238817 conn(0x7f79180f09a0 msgr2=0x7f79180f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:02.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.953+0000 7f791e6d1700 1 -- 172.21.15.49:0/3687238817 shutdown_connections 2023-11-30T21:53:02.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.953+0000 7f791e6d1700 1 -- 172.21.15.49:0/3687238817 wait complete. 2023-11-30T21:53:02.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.953+0000 7f791e6d1700 1 Processor -- start 2023-11-30T21:53:02.956 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.953+0000 7f791e6d1700 1 -- start start 2023-11-30T21:53:02.957 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.953+0000 7f791e6d1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79180601e0 0x7f7918106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:02.957 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.953+0000 7f791e6d1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79181069f0 con 0x7f79180601e0 2023-11-30T21:53:02.958 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.954+0000 7f791d6cf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79180601e0 0x7f7918106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:02.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.954+0000 7f791d6cf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79180601e0 0x7f7918106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:55908/0 (socket says 172.21.15.49:55908) 2023-11-30T21:53:02.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.954+0000 7f791d6cf700 1 -- 172.21.15.49:0/754980881 learned_addr learned my addr 172.21.15.49:0/754980881 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:53:02.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.954+0000 7f791d6cf700 1 -- 172.21.15.49:0/754980881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7914009640 con 0x7f79180601e0 2023-11-30T21:53:02.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.954+0000 7f791d6cf700 1 --2- 172.21.15.49:0/754980881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79180601e0 0x7f7918106500 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f7914004b70 tx=0x7f7914004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:02.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.954+0000 7f790dffb700 1 -- 172.21.15.49:0/754980881 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7914018070 con 0x7f79180601e0 2023-11-30T21:53:02.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.954+0000 7f791e6d1700 1 -- 172.21.15.49:0/754980881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7918106bf0 con 0x7f79180601e0 2023-11-30T21:53:02.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.954+0000 7f790dffb700 1 -- 172.21.15.49:0/754980881 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f791400b870 con 0x7f79180601e0 2023-11-30T21:53:02.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.954+0000 7f790dffb700 1 -- 172.21.15.49:0/754980881 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f791401c560 con 0x7f79180601e0 2023-11-30T21:53:02.959 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.955+0000 7f791e6d1700 1 -- 172.21.15.49:0/754980881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7918107080 con 0x7f79180601e0 2023-11-30T21:53:02.960 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.955+0000 7f791e6d1700 1 -- 172.21.15.49:0/754980881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7918045ad0 con 0x7f79180601e0 2023-11-30T21:53:02.960 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.955+0000 7f790dffb700 1 -- 172.21.15.49:0/754980881 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7914013630 con 0x7f79180601e0 2023-11-30T21:53:02.960 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.955+0000 7f790dffb700 1 --2- 172.21.15.49:0/754980881 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f790402ff60 0x7f7904032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:02.960 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.956+0000 7f790dffb700 1 -- 172.21.15.49:0/754980881 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f79140411d0 con 0x7f79180601e0 2023-11-30T21:53:02.960 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.956+0000 7f791cece700 1 --2- 172.21.15.49:0/754980881 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f790402ff60 0x7f7904032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:02.960 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.956+0000 7f791cece700 1 --2- 172.21.15.49:0/754980881 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f790402ff60 0x7f7904032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f79080096a0 tx=0x7f7908006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:02.961 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:02.960+0000 7f790dffb700 1 -- 172.21.15.49:0/754980881 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7914004810 con 0x7f79180601e0 2023-11-30T21:53:03.177 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:03.176+0000 7f791e6d1700 1 -- 172.21.15.49:0/754980881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7918107ed0 con 0x7f79180601e0 2023-11-30T21:53:03.178 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:03.177+0000 7f790dffb700 1 -- 172.21.15.49:0/754980881 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7914012380 con 0x7f79180601e0 2023-11-30T21:53:03.178 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:53:03.178 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:53:03.180 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:03.179+0000 7f791e6d1700 1 -- 172.21.15.49:0/754980881 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f790402ff60 msgr2=0x7f7904032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:03.180 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:03.179+0000 7f791e6d1700 1 --2- 172.21.15.49:0/754980881 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f790402ff60 0x7f7904032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f79080096a0 tx=0x7f7908006b40).stop 2023-11-30T21:53:03.180 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:03.179+0000 7f791e6d1700 1 -- 172.21.15.49:0/754980881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79180601e0 msgr2=0x7f7918106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:03.180 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:03.179+0000 7f791e6d1700 1 --2- 172.21.15.49:0/754980881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79180601e0 0x7f7918106500 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f7914004b70 tx=0x7f7914004d10).stop 2023-11-30T21:53:03.180 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:03.179+0000 7f791e6d1700 1 -- 172.21.15.49:0/754980881 shutdown_connections 2023-11-30T21:53:03.180 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:03.179+0000 7f791e6d1700 1 --2- 172.21.15.49:0/754980881 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f790402ff60 0x7f7904032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:03.180 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:03.179+0000 7f791e6d1700 1 --2- 172.21.15.49:0/754980881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f79180601e0 0x7f7918106500 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:03.180 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:03.179+0000 7f791e6d1700 1 -- 172.21.15.49:0/754980881 >> 172.21.15.49:0/754980881 conn(0x7f79180f09a0 msgr2=0x7f79180f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:03.180 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:03.179+0000 7f791e6d1700 1 -- 172.21.15.49:0/754980881 shutdown_connections 2023-11-30T21:53:03.181 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:03.179+0000 7f791e6d1700 1 -- 172.21.15.49:0/754980881 wait complete. 2023-11-30T21:53:03.181 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:53:03.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:03 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:01.544670+0000 mgr.smithi027.brrjcl (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:03.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:03 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:53:03.177774+0000 mon.smithi027 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.49:0/754980881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:53:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:53:03.593624+0000 mon.smithi027 (mon.0) 315 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/mirror_snapshot_schedule"}]: dispatch 2023-11-30T21:53:04.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:04 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:53:03.599160+0000 mon.smithi027 (mon.0) 316 : audit [INF] from='mgr.14162 172.21.15.27:0/1931182494' entity='mgr.smithi027.brrjcl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.brrjcl/trash_purge_schedule"}]: dispatch 2023-11-30T21:53:04.610 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:53:04.611 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:53:05.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:05 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:03.545035+0000 mgr.smithi027.brrjcl (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:05.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.862+0000 7fe72f4fe700 1 -- 172.21.15.49:0/606975098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7280f5410 msgr2=0x7fe7280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:05.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.862+0000 7fe72f4fe700 1 --2- 172.21.15.49:0/606975098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7280f5410 0x7fe7280f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fe724009fb0 tx=0x7fe724009b30).stop 2023-11-30T21:53:05.867 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.862+0000 7fe72f4fe700 1 -- 172.21.15.49:0/606975098 shutdown_connections 2023-11-30T21:53:05.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.862+0000 7fe72f4fe700 1 --2- 172.21.15.49:0/606975098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7280f5410 0x7fe7280f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:05.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.862+0000 7fe72f4fe700 1 -- 172.21.15.49:0/606975098 >> 172.21.15.49:0/606975098 conn(0x7fe7280f0b70 msgr2=0x7fe7280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:05.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.862+0000 7fe72f4fe700 1 -- 172.21.15.49:0/606975098 shutdown_connections 2023-11-30T21:53:05.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.862+0000 7fe72f4fe700 1 -- 172.21.15.49:0/606975098 wait complete. 2023-11-30T21:53:05.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.863+0000 7fe72f4fe700 1 Processor -- start 2023-11-30T21:53:05.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.863+0000 7fe72f4fe700 1 -- start start 2023-11-30T21:53:05.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.863+0000 7fe72f4fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7280f5410 0x7fe728109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:05.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.863+0000 7fe72f4fe700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe728060220 con 0x7fe7280f5410 2023-11-30T21:53:05.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.863+0000 7fe72d29a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7280f5410 0x7fe728109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:05.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.863+0000 7fe72d29a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7280f5410 0x7fe728109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:55918/0 (socket says 172.21.15.49:55918) 2023-11-30T21:53:05.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.863+0000 7fe72d29a700 1 -- 172.21.15.49:0/1477906774 learned_addr learned my addr 172.21.15.49:0/1477906774 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:53:05.868 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.864+0000 7fe72d29a700 1 -- 172.21.15.49:0/1477906774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe724005040 con 0x7fe7280f5410 2023-11-30T21:53:05.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.864+0000 7fe72d29a700 1 --2- 172.21.15.49:0/1477906774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7280f5410 0x7fe728109450 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fe724013510 tx=0x7fe72400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:05.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.864+0000 7fe719ffb700 1 -- 172.21.15.49:0/1477906774 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe724009320 con 0x7fe7280f5410 2023-11-30T21:53:05.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.864+0000 7fe72f4fe700 1 -- 172.21.15.49:0/1477906774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe728060420 con 0x7fe7280f5410 2023-11-30T21:53:05.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.864+0000 7fe719ffb700 1 -- 172.21.15.49:0/1477906774 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe724013f00 con 0x7fe7280f5410 2023-11-30T21:53:05.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.864+0000 7fe719ffb700 1 -- 172.21.15.49:0/1477906774 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe724014d90 con 0x7fe7280f5410 2023-11-30T21:53:05.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.864+0000 7fe72f4fe700 1 -- 172.21.15.49:0/1477906774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe728109b00 con 0x7fe7280f5410 2023-11-30T21:53:05.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.865+0000 7fe72f4fe700 1 -- 172.21.15.49:0/1477906774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe728059100 con 0x7fe7280f5410 2023-11-30T21:53:05.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.865+0000 7fe719ffb700 1 -- 172.21.15.49:0/1477906774 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7240095f0 con 0x7fe7280f5410 2023-11-30T21:53:05.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.865+0000 7fe719ffb700 1 --2- 172.21.15.49:0/1477906774 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe71402ff60 0x7fe714032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:05.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.865+0000 7fe719ffb700 1 -- 172.21.15.49:0/1477906774 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe724040fc0 con 0x7fe7280f5410 2023-11-30T21:53:05.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.865+0000 7fe72ca99700 1 --2- 172.21.15.49:0/1477906774 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe71402ff60 0x7fe714032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:05.869 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.866+0000 7fe72ca99700 1 --2- 172.21.15.49:0/1477906774 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe71402ff60 0x7fe714032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fe71c0096a0 tx=0x7fe71c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:05.870 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:05.870+0000 7fe719ffb700 1 -- 172.21.15.49:0/1477906774 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe724014420 con 0x7fe7280f5410 2023-11-30T21:53:06.089 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:06.088+0000 7fe72f4fe700 1 -- 172.21.15.49:0/1477906774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe728045ad0 con 0x7fe7280f5410 2023-11-30T21:53:06.089 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:06.089+0000 7fe719ffb700 1 -- 172.21.15.49:0/1477906774 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe724010230 con 0x7fe7280f5410 2023-11-30T21:53:06.090 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:53:06.090 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:53:06.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:06.090+0000 7fe72f4fe700 1 -- 172.21.15.49:0/1477906774 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe71402ff60 msgr2=0x7fe714032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:06.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:06.090+0000 7fe72f4fe700 1 --2- 172.21.15.49:0/1477906774 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe71402ff60 0x7fe714032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fe71c0096a0 tx=0x7fe71c006b40).stop 2023-11-30T21:53:06.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:06.090+0000 7fe72f4fe700 1 -- 172.21.15.49:0/1477906774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7280f5410 msgr2=0x7fe728109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:06.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:06.090+0000 7fe72f4fe700 1 --2- 172.21.15.49:0/1477906774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7280f5410 0x7fe728109450 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fe724013510 tx=0x7fe72400b040).stop 2023-11-30T21:53:06.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:06.090+0000 7fe72f4fe700 1 -- 172.21.15.49:0/1477906774 shutdown_connections 2023-11-30T21:53:06.091 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:06.090+0000 7fe72f4fe700 1 --2- 172.21.15.49:0/1477906774 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7fe71402ff60 0x7fe714032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:06.092 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:06.090+0000 7fe72f4fe700 1 --2- 172.21.15.49:0/1477906774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7280f5410 0x7fe728109450 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:06.092 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:06.090+0000 7fe72f4fe700 1 -- 172.21.15.49:0/1477906774 >> 172.21.15.49:0/1477906774 conn(0x7fe7280f0b70 msgr2=0x7fe7280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:06.092 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:06.090+0000 7fe72f4fe700 1 -- 172.21.15.49:0/1477906774 shutdown_connections 2023-11-30T21:53:06.092 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:06.090+0000 7fe72f4fe700 1 -- 172.21.15.49:0/1477906774 wait complete. 2023-11-30T21:53:06.092 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:53:06.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:06 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:53:06.089415+0000 mon.smithi027 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.49:0/1477906774' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:53:07.513 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:07 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:05.545332+0000 mgr.smithi027.brrjcl (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:07.537 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:53:07.537 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:53:08.775 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.772+0000 7f21b794f700 1 -- 172.21.15.49:0/3061163047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b00f6510 msgr2=0x7f21b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:08.775 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.772+0000 7f21b794f700 1 --2- 172.21.15.49:0/3061163047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b00f6510 0x7f21b00f6930 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f21ac009fb0 tx=0x7f21ac009b30).stop 2023-11-30T21:53:08.775 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.773+0000 7f21b794f700 1 -- 172.21.15.49:0/3061163047 shutdown_connections 2023-11-30T21:53:08.775 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.773+0000 7f21b794f700 1 --2- 172.21.15.49:0/3061163047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b00f6510 0x7f21b00f6930 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:08.775 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.773+0000 7f21b794f700 1 -- 172.21.15.49:0/3061163047 >> 172.21.15.49:0/3061163047 conn(0x7f21b00f1c70 msgr2=0x7f21b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:08.775 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.773+0000 7f21b794f700 1 -- 172.21.15.49:0/3061163047 shutdown_connections 2023-11-30T21:53:08.775 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.773+0000 7f21b794f700 1 -- 172.21.15.49:0/3061163047 wait complete. 2023-11-30T21:53:08.776 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.773+0000 7f21b794f700 1 Processor -- start 2023-11-30T21:53:08.776 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.773+0000 7f21b794f700 1 -- start start 2023-11-30T21:53:08.776 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.773+0000 7f21b794f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b00f6510 0x7f21b0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:08.776 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.773+0000 7f21b794f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b0106c20 con 0x7f21b00f6510 2023-11-30T21:53:08.777 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.774+0000 7f21b56eb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b00f6510 0x7f21b0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:08.777 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.774+0000 7f21b56eb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b00f6510 0x7f21b0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:60884/0 (socket says 172.21.15.49:60884) 2023-11-30T21:53:08.777 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.774+0000 7f21b56eb700 1 -- 172.21.15.49:0/3954569959 learned_addr learned my addr 172.21.15.49:0/3954569959 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:53:08.777 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.774+0000 7f21b56eb700 1 -- 172.21.15.49:0/3954569959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac005040 con 0x7f21b00f6510 2023-11-30T21:53:08.777 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.774+0000 7f21b56eb700 1 --2- 172.21.15.49:0/3954569959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b00f6510 0x7f21b0106730 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f21ac013510 tx=0x7f21ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:08.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.774+0000 7f21a1ffb700 1 -- 172.21.15.49:0/3954569959 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21ac009320 con 0x7f21b00f6510 2023-11-30T21:53:08.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.774+0000 7f21b794f700 1 -- 172.21.15.49:0/3954569959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b0106e20 con 0x7f21b00f6510 2023-11-30T21:53:08.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.774+0000 7f21a1ffb700 1 -- 172.21.15.49:0/3954569959 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21ac013f00 con 0x7f21b00f6510 2023-11-30T21:53:08.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.774+0000 7f21a1ffb700 1 -- 172.21.15.49:0/3954569959 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21ac014d90 con 0x7f21b00f6510 2023-11-30T21:53:08.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.775+0000 7f21b794f700 1 -- 172.21.15.49:0/3954569959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b01072b0 con 0x7f21b00f6510 2023-11-30T21:53:08.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.775+0000 7f21a1ffb700 1 -- 172.21.15.49:0/3954569959 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21ac0095f0 con 0x7f21b00f6510 2023-11-30T21:53:08.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.775+0000 7f21b794f700 1 -- 172.21.15.49:0/3954569959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b0045ad0 con 0x7f21b00f6510 2023-11-30T21:53:08.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.775+0000 7f21a1ffb700 1 --2- 172.21.15.49:0/3954569959 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f219c02ff60 0x7f219c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:08.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.776+0000 7f21a1ffb700 1 -- 172.21.15.49:0/3954569959 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f21ac041000 con 0x7f21b00f6510 2023-11-30T21:53:08.778 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.776+0000 7f21b4eea700 1 --2- 172.21.15.49:0/3954569959 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f219c02ff60 0x7f219c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:08.779 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.776+0000 7f21b4eea700 1 --2- 172.21.15.49:0/3954569959 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f219c02ff60 0x7f219c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f21a40096a0 tx=0x7f21a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:08.782 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.780+0000 7f21a1ffb700 1 -- 172.21.15.49:0/3954569959 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21ac014420 con 0x7f21b00f6510 2023-11-30T21:53:09.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.998+0000 7f21b794f700 1 -- 172.21.15.49:0/3954569959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21b00fbf60 con 0x7f21b00f6510 2023-11-30T21:53:09.000 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:08.999+0000 7f21a1ffb700 1 -- 172.21.15.49:0/3954569959 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f21ac01d7f0 con 0x7f21b00f6510 2023-11-30T21:53:09.001 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:53:09.001 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:53:09.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:09.000+0000 7f21b794f700 1 -- 172.21.15.49:0/3954569959 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f219c02ff60 msgr2=0x7f219c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:09.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:09.000+0000 7f21b794f700 1 --2- 172.21.15.49:0/3954569959 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f219c02ff60 0x7f219c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f21a40096a0 tx=0x7f21a4006b40).stop 2023-11-30T21:53:09.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:09.000+0000 7f21b794f700 1 -- 172.21.15.49:0/3954569959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b00f6510 msgr2=0x7f21b0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:09.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:09.000+0000 7f21b794f700 1 --2- 172.21.15.49:0/3954569959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b00f6510 0x7f21b0106730 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f21ac013510 tx=0x7f21ac00b040).stop 2023-11-30T21:53:09.002 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:09.000+0000 7f21b794f700 1 -- 172.21.15.49:0/3954569959 shutdown_connections 2023-11-30T21:53:09.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:09.001+0000 7f21b794f700 1 --2- 172.21.15.49:0/3954569959 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f219c02ff60 0x7f219c032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:09.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:09.001+0000 7f21b794f700 1 --2- 172.21.15.49:0/3954569959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21b00f6510 0x7f21b0106730 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:09.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:09.001+0000 7f21b794f700 1 -- 172.21.15.49:0/3954569959 >> 172.21.15.49:0/3954569959 conn(0x7f21b00f1c70 msgr2=0x7f21b00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:09.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:09.001+0000 7f21b794f700 1 -- 172.21.15.49:0/3954569959 shutdown_connections 2023-11-30T21:53:09.003 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:09.001+0000 7f21b794f700 1 -- 172.21.15.49:0/3954569959 wait complete. 2023-11-30T21:53:09.003 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:53:09.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:09 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:07.545601+0000 mgr.smithi027.brrjcl (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:09.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:09 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:53:09.000545+0000 mon.smithi027 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.49:0/3954569959' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:53:10.443 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:53:10.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:53:11.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:11 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:09.545908+0000 mgr.smithi027.brrjcl (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:11.714 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.711+0000 7f31ce853700 1 -- 172.21.15.49:0/3063460334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c80f5410 msgr2=0x7f31c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:11.714 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.711+0000 7f31ce853700 1 --2- 172.21.15.49:0/3063460334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c80f5410 0x7f31c80f5830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f31b0009fb0 tx=0x7f31b0009b30).stop 2023-11-30T21:53:11.714 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.711+0000 7f31ce853700 1 -- 172.21.15.49:0/3063460334 shutdown_connections 2023-11-30T21:53:11.714 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.711+0000 7f31ce853700 1 --2- 172.21.15.49:0/3063460334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c80f5410 0x7f31c80f5830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:11.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.711+0000 7f31ce853700 1 -- 172.21.15.49:0/3063460334 >> 172.21.15.49:0/3063460334 conn(0x7f31c80f0b70 msgr2=0x7f31c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:11.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.711+0000 7f31ce853700 1 -- 172.21.15.49:0/3063460334 shutdown_connections 2023-11-30T21:53:11.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.711+0000 7f31ce853700 1 -- 172.21.15.49:0/3063460334 wait complete. 2023-11-30T21:53:11.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.712+0000 7f31ce853700 1 Processor -- start 2023-11-30T21:53:11.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.712+0000 7f31ce853700 1 -- start start 2023-11-30T21:53:11.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.712+0000 7f31ce853700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c80f5410 0x7f31c8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:11.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.712+0000 7f31ce853700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c8106c20 con 0x7f31c80f5410 2023-11-30T21:53:11.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.712+0000 7f31c7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c80f5410 0x7f31c8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:11.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.712+0000 7f31c7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c80f5410 0x7f31c8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:60894/0 (socket says 172.21.15.49:60894) 2023-11-30T21:53:11.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.712+0000 7f31c7fff700 1 -- 172.21.15.49:0/2561010541 learned_addr learned my addr 172.21.15.49:0/2561010541 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:53:11.715 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.713+0000 7f31c7fff700 1 -- 172.21.15.49:0/2561010541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b0005040 con 0x7f31c80f5410 2023-11-30T21:53:11.716 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.713+0000 7f31c7fff700 1 --2- 172.21.15.49:0/2561010541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c80f5410 0x7f31c8106730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f31b0013510 tx=0x7f31b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:11.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.713+0000 7f31c4ff9700 1 -- 172.21.15.49:0/2561010541 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31b0009320 con 0x7f31c80f5410 2023-11-30T21:53:11.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.713+0000 7f31c4ff9700 1 -- 172.21.15.49:0/2561010541 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31b0013f00 con 0x7f31c80f5410 2023-11-30T21:53:11.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.713+0000 7f31ce853700 1 -- 172.21.15.49:0/2561010541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c8106e20 con 0x7f31c80f5410 2023-11-30T21:53:11.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.713+0000 7f31c4ff9700 1 -- 172.21.15.49:0/2561010541 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31b0014d90 con 0x7f31c80f5410 2023-11-30T21:53:11.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.713+0000 7f31ce853700 1 -- 172.21.15.49:0/2561010541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c81072b0 con 0x7f31c80f5410 2023-11-30T21:53:11.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.714+0000 7f31ce853700 1 -- 172.21.15.49:0/2561010541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c8045ad0 con 0x7f31c80f5410 2023-11-30T21:53:11.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.714+0000 7f31c4ff9700 1 -- 172.21.15.49:0/2561010541 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31b00095f0 con 0x7f31c80f5410 2023-11-30T21:53:11.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.714+0000 7f31c4ff9700 1 --2- 172.21.15.49:0/2561010541 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f31b402ff60 0x7f31b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:11.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.714+0000 7f31c4ff9700 1 -- 172.21.15.49:0/2561010541 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f31b00410c0 con 0x7f31c80f5410 2023-11-30T21:53:11.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.714+0000 7f31c77fe700 1 --2- 172.21.15.49:0/2561010541 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f31b402ff60 0x7f31b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:11.717 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.715+0000 7f31c77fe700 1 --2- 172.21.15.49:0/2561010541 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f31b402ff60 0x7f31b4032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f31b80096a0 tx=0x7f31b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:11.721 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.719+0000 7f31c4ff9700 1 -- 172.21.15.49:0/2561010541 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31b0014420 con 0x7f31c80f5410 2023-11-30T21:53:11.939 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.937+0000 7f31ce853700 1 -- 172.21.15.49:0/2561010541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31c81080a0 con 0x7f31c80f5410 2023-11-30T21:53:11.939 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.938+0000 7f31c4ff9700 1 -- 172.21.15.49:0/2561010541 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f31b0010230 con 0x7f31c80f5410 2023-11-30T21:53:11.940 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:53:11.940 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:53:11.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.939+0000 7f31ce853700 1 -- 172.21.15.49:0/2561010541 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f31b402ff60 msgr2=0x7f31b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:11.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.939+0000 7f31ce853700 1 --2- 172.21.15.49:0/2561010541 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f31b402ff60 0x7f31b4032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f31b80096a0 tx=0x7f31b8006b40).stop 2023-11-30T21:53:11.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.939+0000 7f31ce853700 1 -- 172.21.15.49:0/2561010541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c80f5410 msgr2=0x7f31c8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:11.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.939+0000 7f31ce853700 1 --2- 172.21.15.49:0/2561010541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c80f5410 0x7f31c8106730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f31b0013510 tx=0x7f31b000b040).stop 2023-11-30T21:53:11.941 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.939+0000 7f31ce853700 1 -- 172.21.15.49:0/2561010541 shutdown_connections 2023-11-30T21:53:11.942 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.939+0000 7f31ce853700 1 --2- 172.21.15.49:0/2561010541 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f31b402ff60 0x7f31b4032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:11.942 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.939+0000 7f31ce853700 1 --2- 172.21.15.49:0/2561010541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31c80f5410 0x7f31c8106730 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:11.942 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.939+0000 7f31ce853700 1 -- 172.21.15.49:0/2561010541 >> 172.21.15.49:0/2561010541 conn(0x7f31c80f0b70 msgr2=0x7f31c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:11.942 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.939+0000 7f31ce853700 1 -- 172.21.15.49:0/2561010541 shutdown_connections 2023-11-30T21:53:11.942 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:11.940+0000 7f31ce853700 1 -- 172.21.15.49:0/2561010541 wait complete. 2023-11-30T21:53:11.942 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:53:12.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:12 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:53:11.939396+0000 mon.smithi027 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.49:0/2561010541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:53:13.348 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:53:13.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:53:13.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:13 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:11.546247+0000 mgr.smithi027.brrjcl (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:14.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.566+0000 7feabd759700 1 -- 172.21.15.49:0/3477897248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feab80f7b60 msgr2=0x7feab80f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:14.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.566+0000 7feabd759700 1 --2- 172.21.15.49:0/3477897248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feab80f7b60 0x7feab80f7f80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7feaa8009fb0 tx=0x7feaa8009b30).stop 2023-11-30T21:53:14.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.566+0000 7feabd759700 1 -- 172.21.15.49:0/3477897248 shutdown_connections 2023-11-30T21:53:14.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.566+0000 7feabd759700 1 --2- 172.21.15.49:0/3477897248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feab80f7b60 0x7feab80f7f80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:14.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.566+0000 7feabd759700 1 -- 172.21.15.49:0/3477897248 >> 172.21.15.49:0/3477897248 conn(0x7feab80f02e0 msgr2=0x7feab80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:14.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.566+0000 7feabd759700 1 -- 172.21.15.49:0/3477897248 shutdown_connections 2023-11-30T21:53:14.568 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.566+0000 7feabd759700 1 -- 172.21.15.49:0/3477897248 wait complete. 2023-11-30T21:53:14.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.567+0000 7feabd759700 1 Processor -- start 2023-11-30T21:53:14.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.567+0000 7feabd759700 1 -- start start 2023-11-30T21:53:14.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.567+0000 7feabd759700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feab80f7b60 0x7feab8106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:14.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.567+0000 7feabd759700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab8106c30 con 0x7feab80f7b60 2023-11-30T21:53:14.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.567+0000 7feab6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feab80f7b60 0x7feab8106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:14.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.567+0000 7feab6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feab80f7b60 0x7feab8106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:60910/0 (socket says 172.21.15.49:60910) 2023-11-30T21:53:14.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.567+0000 7feab6ffd700 1 -- 172.21.15.49:0/2844558648 learned_addr learned my addr 172.21.15.49:0/2844558648 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:53:14.570 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.568+0000 7feab6ffd700 1 -- 172.21.15.49:0/2844558648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa8005040 con 0x7feab80f7b60 2023-11-30T21:53:14.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.568+0000 7feab6ffd700 1 --2- 172.21.15.49:0/2844558648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feab80f7b60 0x7feab8106740 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7feaa8013510 tx=0x7feaa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:14.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.568+0000 7fea9f7fe700 1 -- 172.21.15.49:0/2844558648 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feaa8009320 con 0x7feab80f7b60 2023-11-30T21:53:14.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.568+0000 7feabd759700 1 -- 172.21.15.49:0/2844558648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab8106e30 con 0x7feab80f7b60 2023-11-30T21:53:14.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.568+0000 7fea9f7fe700 1 -- 172.21.15.49:0/2844558648 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feaa8013f00 con 0x7feab80f7b60 2023-11-30T21:53:14.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.568+0000 7fea9f7fe700 1 -- 172.21.15.49:0/2844558648 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feaa8014d90 con 0x7feab80f7b60 2023-11-30T21:53:14.571 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.568+0000 7feabd759700 1 -- 172.21.15.49:0/2844558648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab81072c0 con 0x7feab80f7b60 2023-11-30T21:53:14.572 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.569+0000 7fea9f7fe700 1 -- 172.21.15.49:0/2844558648 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feaa80095f0 con 0x7feab80f7b60 2023-11-30T21:53:14.572 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.569+0000 7feabd759700 1 -- 172.21.15.49:0/2844558648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab8045ad0 con 0x7feab80f7b60 2023-11-30T21:53:14.572 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.569+0000 7fea9f7fe700 1 --2- 172.21.15.49:0/2844558648 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feaa002ff60 0x7feaa0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:14.572 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.569+0000 7fea9f7fe700 1 -- 172.21.15.49:0/2844558648 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feaa8041080 con 0x7feab80f7b60 2023-11-30T21:53:14.572 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.569+0000 7feab67fc700 1 --2- 172.21.15.49:0/2844558648 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feaa002ff60 0x7feaa0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:14.572 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.570+0000 7feab67fc700 1 --2- 172.21.15.49:0/2844558648 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feaa002ff60 0x7feaa0032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7feaac0096a0 tx=0x7feaac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:14.575 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.574+0000 7fea9f7fe700 1 -- 172.21.15.49:0/2844558648 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feaa8014420 con 0x7feab80f7b60 2023-11-30T21:53:14.794 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.793+0000 7feabd759700 1 -- 172.21.15.49:0/2844558648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feab8108110 con 0x7feab80f7b60 2023-11-30T21:53:14.795 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.793+0000 7fea9f7fe700 1 -- 172.21.15.49:0/2844558648 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feaa8010230 con 0x7feab80f7b60 2023-11-30T21:53:14.795 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:53:14.795 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:53:14.797 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.795+0000 7feabd759700 1 -- 172.21.15.49:0/2844558648 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feaa002ff60 msgr2=0x7feaa0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:14.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.795+0000 7feabd759700 1 --2- 172.21.15.49:0/2844558648 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feaa002ff60 0x7feaa0032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7feaac0096a0 tx=0x7feaac006b40).stop 2023-11-30T21:53:14.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.795+0000 7feabd759700 1 -- 172.21.15.49:0/2844558648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feab80f7b60 msgr2=0x7feab8106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:14.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.795+0000 7feabd759700 1 --2- 172.21.15.49:0/2844558648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feab80f7b60 0x7feab8106740 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7feaa8013510 tx=0x7feaa800b040).stop 2023-11-30T21:53:14.798 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.795+0000 7feabd759700 1 -- 172.21.15.49:0/2844558648 shutdown_connections 2023-11-30T21:53:14.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.795+0000 7feabd759700 1 --2- 172.21.15.49:0/2844558648 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7feaa002ff60 0x7feaa0032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:14.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.795+0000 7feabd759700 1 --2- 172.21.15.49:0/2844558648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feab80f7b60 0x7feab8106740 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:14.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.795+0000 7feabd759700 1 -- 172.21.15.49:0/2844558648 >> 172.21.15.49:0/2844558648 conn(0x7feab80f02e0 msgr2=0x7feab80f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:14.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.795+0000 7feabd759700 1 -- 172.21.15.49:0/2844558648 shutdown_connections 2023-11-30T21:53:14.799 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:14.795+0000 7feabd759700 1 -- 172.21.15.49:0/2844558648 wait complete. 2023-11-30T21:53:14.800 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:53:15.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:15 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:13.546527+0000 mgr.smithi027.brrjcl (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:15.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:15 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:53:14.794908+0000 mon.smithi027 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.49:0/2844558648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:53:16.250 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:53:16.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:53:17.479 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.477+0000 7f43211a8700 1 -- 172.21.15.49:0/4138015485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f431c0f5410 msgr2=0x7f431c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.477+0000 7f43211a8700 1 --2- 172.21.15.49:0/4138015485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f431c0f5410 0x7f431c0f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f430c009fb0 tx=0x7f430c009b30).stop 2023-11-30T21:53:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.477+0000 7f43211a8700 1 -- 172.21.15.49:0/4138015485 shutdown_connections 2023-11-30T21:53:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.477+0000 7f43211a8700 1 --2- 172.21.15.49:0/4138015485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f431c0f5410 0x7f431c0f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.477+0000 7f43211a8700 1 -- 172.21.15.49:0/4138015485 >> 172.21.15.49:0/4138015485 conn(0x7f431c0f0b70 msgr2=0x7f431c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:17.481 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.477+0000 7f43211a8700 1 -- 172.21.15.49:0/4138015485 shutdown_connections 2023-11-30T21:53:17.481 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.477+0000 7f43211a8700 1 -- 172.21.15.49:0/4138015485 wait complete. 2023-11-30T21:53:17.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.478+0000 7f43211a8700 1 Processor -- start 2023-11-30T21:53:17.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.478+0000 7f43211a8700 1 -- start start 2023-11-30T21:53:17.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.478+0000 7f43211a8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f431c0f5410 0x7f431c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:17.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.478+0000 7f43211a8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c0fb820 con 0x7f431c0f5410 2023-11-30T21:53:17.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.479+0000 7f431ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f431c0f5410 0x7f431c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:17.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.479+0000 7f431ad9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f431c0f5410 0x7f431c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:60932/0 (socket says 172.21.15.49:60932) 2023-11-30T21:53:17.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.479+0000 7f431ad9d700 1 -- 172.21.15.49:0/1030369402 learned_addr learned my addr 172.21.15.49:0/1030369402 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:53:17.483 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.479+0000 7f431ad9d700 1 -- 172.21.15.49:0/1030369402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c005040 con 0x7f431c0f5410 2023-11-30T21:53:17.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.479+0000 7f431ad9d700 1 --2- 172.21.15.49:0/1030369402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f431c0f5410 0x7f431c0fd3a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f430c013510 tx=0x7f430c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:17.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.479+0000 7f43037fe700 1 -- 172.21.15.49:0/1030369402 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f430c009320 con 0x7f431c0f5410 2023-11-30T21:53:17.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.479+0000 7f43037fe700 1 -- 172.21.15.49:0/1030369402 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f430c013f00 con 0x7f431c0f5410 2023-11-30T21:53:17.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.479+0000 7f43211a8700 1 -- 172.21.15.49:0/1030369402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431c0fb9c0 con 0x7f431c0f5410 2023-11-30T21:53:17.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.479+0000 7f43037fe700 1 -- 172.21.15.49:0/1030369402 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f430c014d90 con 0x7f431c0f5410 2023-11-30T21:53:17.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.479+0000 7f43211a8700 1 -- 172.21.15.49:0/1030369402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431c0fbe50 con 0x7f431c0f5410 2023-11-30T21:53:17.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.480+0000 7f43037fe700 1 -- 172.21.15.49:0/1030369402 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f430c014420 con 0x7f431c0f5410 2023-11-30T21:53:17.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.480+0000 7f43211a8700 1 -- 172.21.15.49:0/1030369402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431c059100 con 0x7f431c0f5410 2023-11-30T21:53:17.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.480+0000 7f43037fe700 1 --2- 172.21.15.49:0/1030369402 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f430402ffb0 0x7f4304032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:17.484 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.480+0000 7f43037fe700 1 -- 172.21.15.49:0/1030369402 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f430c041090 con 0x7f431c0f5410 2023-11-30T21:53:17.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.481+0000 7f431a59c700 1 --2- 172.21.15.49:0/1030369402 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f430402ffb0 0x7f4304032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:17.485 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.481+0000 7f431a59c700 1 --2- 172.21.15.49:0/1030369402 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f430402ffb0 0x7f4304032470 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f43100096a0 tx=0x7f4310006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:17.487 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.485+0000 7f43037fe700 1 -- 172.21.15.49:0/1030369402 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f430c06d2b0 con 0x7f431c0f5410 2023-11-30T21:53:17.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:17 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:15.546887+0000 mgr.smithi027.brrjcl (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:17.707 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.706+0000 7f43211a8700 1 -- 172.21.15.49:0/1030369402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f431c0fae60 con 0x7f431c0f5410 2023-11-30T21:53:17.708 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.707+0000 7f43037fe700 1 -- 172.21.15.49:0/1030369402 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f430c0095f0 con 0x7f431c0f5410 2023-11-30T21:53:17.708 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:53:17.708 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:53:17.710 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.708+0000 7f43211a8700 1 -- 172.21.15.49:0/1030369402 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f430402ffb0 msgr2=0x7f4304032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.708+0000 7f43211a8700 1 --2- 172.21.15.49:0/1030369402 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f430402ffb0 0x7f4304032470 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f43100096a0 tx=0x7f4310006b40).stop 2023-11-30T21:53:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.708+0000 7f43211a8700 1 -- 172.21.15.49:0/1030369402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f431c0f5410 msgr2=0x7f431c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.708+0000 7f43211a8700 1 --2- 172.21.15.49:0/1030369402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f431c0f5410 0x7f431c0fd3a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f430c013510 tx=0x7f430c00b040).stop 2023-11-30T21:53:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.708+0000 7f43211a8700 1 -- 172.21.15.49:0/1030369402 shutdown_connections 2023-11-30T21:53:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.708+0000 7f43211a8700 1 --2- 172.21.15.49:0/1030369402 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f430402ffb0 0x7f4304032470 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.708+0000 7f43211a8700 1 --2- 172.21.15.49:0/1030369402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f431c0f5410 0x7f431c0fd3a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.708+0000 7f43211a8700 1 -- 172.21.15.49:0/1030369402 >> 172.21.15.49:0/1030369402 conn(0x7f431c0f0b70 msgr2=0x7f431c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:17.711 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.708+0000 7f43211a8700 1 -- 172.21.15.49:0/1030369402 shutdown_connections 2023-11-30T21:53:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:17.708+0000 7f43211a8700 1 -- 172.21.15.49:0/1030369402 wait complete. 2023-11-30T21:53:17.712 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:53:18.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:18 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:53:17.708161+0000 mon.smithi027 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.49:0/1030369402' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:53:19.161 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:53:19.161 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:53:19.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:19 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:17.547268+0000 mgr.smithi027.brrjcl (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:20.446 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.444+0000 7f61fc8c8700 1 -- 172.21.15.49:0/1372194269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61f80f6510 msgr2=0x7f61f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:20.446 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.444+0000 7f61fc8c8700 1 --2- 172.21.15.49:0/1372194269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61f80f6510 0x7f61f80f6930 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f61e8009fb0 tx=0x7f61e8009b30).stop 2023-11-30T21:53:20.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.444+0000 7f61fc8c8700 1 -- 172.21.15.49:0/1372194269 shutdown_connections 2023-11-30T21:53:20.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.444+0000 7f61fc8c8700 1 --2- 172.21.15.49:0/1372194269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61f80f6510 0x7f61f80f6930 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:20.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.444+0000 7f61fc8c8700 1 -- 172.21.15.49:0/1372194269 >> 172.21.15.49:0/1372194269 conn(0x7f61f80f1c70 msgr2=0x7f61f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:20.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.444+0000 7f61fc8c8700 1 -- 172.21.15.49:0/1372194269 shutdown_connections 2023-11-30T21:53:20.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.444+0000 7f61fc8c8700 1 -- 172.21.15.49:0/1372194269 wait complete. 2023-11-30T21:53:20.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.444+0000 7f61fc8c8700 1 Processor -- start 2023-11-30T21:53:20.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.444+0000 7f61fc8c8700 1 -- start start 2023-11-30T21:53:20.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.445+0000 7f61fc8c8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61f80f6510 0x7f61f806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:20.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.445+0000 7f61fc8c8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f8069e20 con 0x7f61f80f6510 2023-11-30T21:53:20.447 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.445+0000 7f61f659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61f80f6510 0x7f61f806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:20.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.445+0000 7f61f659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61f80f6510 0x7f61f806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:50532/0 (socket says 172.21.15.49:50532) 2023-11-30T21:53:20.448 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.445+0000 7f61f659c700 1 -- 172.21.15.49:0/875836736 learned_addr learned my addr 172.21.15.49:0/875836736 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:53:20.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.445+0000 7f61f659c700 1 -- 172.21.15.49:0/875836736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e8005040 con 0x7f61f80f6510 2023-11-30T21:53:20.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.445+0000 7f61f659c700 1 --2- 172.21.15.49:0/875836736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61f80f6510 0x7f61f806b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f61e8013510 tx=0x7f61e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:20.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.445+0000 7f61e6ffd700 1 -- 172.21.15.49:0/875836736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61e8009320 con 0x7f61f80f6510 2023-11-30T21:53:20.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.445+0000 7f61fc8c8700 1 -- 172.21.15.49:0/875836736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f8069fc0 con 0x7f61f80f6510 2023-11-30T21:53:20.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.445+0000 7f61e6ffd700 1 -- 172.21.15.49:0/875836736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61e8013f00 con 0x7f61f80f6510 2023-11-30T21:53:20.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.445+0000 7f61e6ffd700 1 -- 172.21.15.49:0/875836736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61e8014d90 con 0x7f61f80f6510 2023-11-30T21:53:20.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.446+0000 7f61fc8c8700 1 -- 172.21.15.49:0/875836736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f806a450 con 0x7f61f80f6510 2023-11-30T21:53:20.449 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.446+0000 7f61e6ffd700 1 -- 172.21.15.49:0/875836736 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61e80095f0 con 0x7f61f80f6510 2023-11-30T21:53:20.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.446+0000 7f61fc8c8700 1 -- 172.21.15.49:0/875836736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f8059100 con 0x7f61f80f6510 2023-11-30T21:53:20.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.446+0000 7f61e6ffd700 1 --2- 172.21.15.49:0/875836736 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f61e002ff60 0x7f61e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:20.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.447+0000 7f61e6ffd700 1 -- 172.21.15.49:0/875836736 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61e8041000 con 0x7f61f80f6510 2023-11-30T21:53:20.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.447+0000 7f61f5d9b700 1 --2- 172.21.15.49:0/875836736 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f61e002ff60 0x7f61e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:20.450 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.447+0000 7f61f5d9b700 1 --2- 172.21.15.49:0/875836736 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f61e002ff60 0x7f61e0032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f61ec0096a0 tx=0x7f61ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:20.453 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.452+0000 7f61e6ffd700 1 -- 172.21.15.49:0/875836736 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61e8014420 con 0x7f61f80f6510 2023-11-30T21:53:20.678 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.674+0000 7f61fc8c8700 1 -- 172.21.15.49:0/875836736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61f80fbf60 con 0x7f61f80f6510 2023-11-30T21:53:20.678 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.674+0000 7f61e6ffd700 1 -- 172.21.15.49:0/875836736 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f61e801d7f0 con 0x7f61f80f6510 2023-11-30T21:53:20.678 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:53:20.679 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:53:20.680 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.676+0000 7f61fc8c8700 1 -- 172.21.15.49:0/875836736 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f61e002ff60 msgr2=0x7f61e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:20.680 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.676+0000 7f61fc8c8700 1 --2- 172.21.15.49:0/875836736 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f61e002ff60 0x7f61e0032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f61ec0096a0 tx=0x7f61ec006b40).stop 2023-11-30T21:53:20.680 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.676+0000 7f61fc8c8700 1 -- 172.21.15.49:0/875836736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61f80f6510 msgr2=0x7f61f806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:20.680 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.676+0000 7f61fc8c8700 1 --2- 172.21.15.49:0/875836736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61f80f6510 0x7f61f806b9a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f61e8013510 tx=0x7f61e800b040).stop 2023-11-30T21:53:20.680 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.676+0000 7f61fc8c8700 1 -- 172.21.15.49:0/875836736 shutdown_connections 2023-11-30T21:53:20.681 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.676+0000 7f61fc8c8700 1 --2- 172.21.15.49:0/875836736 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f61e002ff60 0x7f61e0032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:20.681 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.676+0000 7f61fc8c8700 1 --2- 172.21.15.49:0/875836736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f61f80f6510 0x7f61f806b9a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:20.681 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.676+0000 7f61fc8c8700 1 -- 172.21.15.49:0/875836736 >> 172.21.15.49:0/875836736 conn(0x7f61f80f1c70 msgr2=0x7f61f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:20.681 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.676+0000 7f61fc8c8700 1 -- 172.21.15.49:0/875836736 shutdown_connections 2023-11-30T21:53:20.681 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:20.676+0000 7f61fc8c8700 1 -- 172.21.15.49:0/875836736 wait complete. 2023-11-30T21:53:20.681 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:53:21.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:21 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:19.547515+0000 mgr.smithi027.brrjcl (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:21.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:21 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:53:20.675822+0000 mon.smithi027 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.49:0/875836736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:53:22.014 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:53:22.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:53:23.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.278+0000 7f3060913700 1 -- 172.21.15.49:0/1380341750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305c0f6510 msgr2=0x7f305c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:23.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.278+0000 7f3060913700 1 --2- 172.21.15.49:0/1380341750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305c0f6510 0x7f305c0f6930 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f304c009fb0 tx=0x7f304c009b30).stop 2023-11-30T21:53:23.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.278+0000 7f3060913700 1 -- 172.21.15.49:0/1380341750 shutdown_connections 2023-11-30T21:53:23.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.278+0000 7f3060913700 1 --2- 172.21.15.49:0/1380341750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305c0f6510 0x7f305c0f6930 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:23.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.278+0000 7f3060913700 1 -- 172.21.15.49:0/1380341750 >> 172.21.15.49:0/1380341750 conn(0x7f305c0f1c70 msgr2=0x7f305c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:23.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.278+0000 7f3060913700 1 -- 172.21.15.49:0/1380341750 shutdown_connections 2023-11-30T21:53:23.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.278+0000 7f3060913700 1 -- 172.21.15.49:0/1380341750 wait complete. 2023-11-30T21:53:23.281 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.279+0000 7f3060913700 1 Processor -- start 2023-11-30T21:53:23.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.279+0000 7f3060913700 1 -- start start 2023-11-30T21:53:23.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.279+0000 7f3060913700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305c0f6510 0x7f305c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:23.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.279+0000 7f3060913700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c106c20 con 0x7f305c0f6510 2023-11-30T21:53:23.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.279+0000 7f305a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305c0f6510 0x7f305c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:23.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.279+0000 7f305a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305c0f6510 0x7f305c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:50554/0 (socket says 172.21.15.49:50554) 2023-11-30T21:53:23.282 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.279+0000 7f305a59c700 1 -- 172.21.15.49:0/2607848465 learned_addr learned my addr 172.21.15.49:0/2607848465 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:53:23.283 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.280+0000 7f305a59c700 1 -- 172.21.15.49:0/2607848465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c005040 con 0x7f305c0f6510 2023-11-30T21:53:23.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.280+0000 7f305a59c700 1 --2- 172.21.15.49:0/2607848465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305c0f6510 0x7f305c106730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f304c013510 tx=0x7f304c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:23.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.280+0000 7f304affd700 1 -- 172.21.15.49:0/2607848465 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f304c009320 con 0x7f305c0f6510 2023-11-30T21:53:23.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.280+0000 7f304affd700 1 -- 172.21.15.49:0/2607848465 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f304c013f00 con 0x7f305c0f6510 2023-11-30T21:53:23.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.280+0000 7f3060913700 1 -- 172.21.15.49:0/2607848465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305c106e20 con 0x7f305c0f6510 2023-11-30T21:53:23.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.280+0000 7f304affd700 1 -- 172.21.15.49:0/2607848465 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f304c014d90 con 0x7f305c0f6510 2023-11-30T21:53:23.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.280+0000 7f3060913700 1 -- 172.21.15.49:0/2607848465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305c1072b0 con 0x7f305c0f6510 2023-11-30T21:53:23.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.281+0000 7f304affd700 1 -- 172.21.15.49:0/2607848465 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f304c0095f0 con 0x7f305c0f6510 2023-11-30T21:53:23.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.281+0000 7f3060913700 1 -- 172.21.15.49:0/2607848465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f305c045ad0 con 0x7f305c0f6510 2023-11-30T21:53:23.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.281+0000 7f304affd700 1 --2- 172.21.15.49:0/2607848465 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f304402ff60 0x7f3044032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:23.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.281+0000 7f304affd700 1 -- 172.21.15.49:0/2607848465 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f304c040fe0 con 0x7f305c0f6510 2023-11-30T21:53:23.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.281+0000 7f3059d9b700 1 --2- 172.21.15.49:0/2607848465 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f304402ff60 0x7f3044032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:23.284 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.282+0000 7f3059d9b700 1 --2- 172.21.15.49:0/2607848465 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f304402ff60 0x7f3044032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f30500096a0 tx=0x7f3050006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:23.287 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.286+0000 7f304affd700 1 -- 172.21.15.49:0/2607848465 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f304c01d7f0 con 0x7f305c0f6510 2023-11-30T21:53:23.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.502+0000 7f3060913700 1 -- 172.21.15.49:0/2607848465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f305c0fbf60 con 0x7f305c0f6510 2023-11-30T21:53:23.504 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.503+0000 7f304affd700 1 -- 172.21.15.49:0/2607848465 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f304c010230 con 0x7f305c0f6510 2023-11-30T21:53:23.505 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:53:23.505 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:53:23.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.504+0000 7f3060913700 1 -- 172.21.15.49:0/2607848465 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f304402ff60 msgr2=0x7f3044032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:23.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.504+0000 7f3060913700 1 --2- 172.21.15.49:0/2607848465 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f304402ff60 0x7f3044032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f30500096a0 tx=0x7f3050006b40).stop 2023-11-30T21:53:23.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.504+0000 7f3060913700 1 -- 172.21.15.49:0/2607848465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305c0f6510 msgr2=0x7f305c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:23.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.504+0000 7f3060913700 1 --2- 172.21.15.49:0/2607848465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305c0f6510 0x7f305c106730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f304c013510 tx=0x7f304c00b040).stop 2023-11-30T21:53:23.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.504+0000 7f3060913700 1 -- 172.21.15.49:0/2607848465 shutdown_connections 2023-11-30T21:53:23.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.504+0000 7f3060913700 1 --2- 172.21.15.49:0/2607848465 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f304402ff60 0x7f3044032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:23.506 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.504+0000 7f3060913700 1 --2- 172.21.15.49:0/2607848465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f305c0f6510 0x7f305c106730 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:23.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.504+0000 7f3060913700 1 -- 172.21.15.49:0/2607848465 >> 172.21.15.49:0/2607848465 conn(0x7f305c0f1c70 msgr2=0x7f305c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:23.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.504+0000 7f3060913700 1 -- 172.21.15.49:0/2607848465 shutdown_connections 2023-11-30T21:53:23.507 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:23.504+0000 7f3060913700 1 -- 172.21.15.49:0/2607848465 wait complete. 2023-11-30T21:53:23.507 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:53:23.510 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:23 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:21.547847+0000 mgr.smithi027.brrjcl (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:24.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:24 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:53:23.504326+0000 mon.smithi027 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.49:0/2607848465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:53:24.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:53:24.928 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:53:25.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:25 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:23.548166+0000 mgr.smithi027.brrjcl (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:26.072 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.069+0000 7f22dcc0c700 1 -- 172.21.15.49:0/741015547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22d80f62a0 msgr2=0x7f22d80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:26.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.069+0000 7f22dcc0c700 1 --2- 172.21.15.49:0/741015547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22d80f62a0 0x7f22d80f66c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f22cc009fb0 tx=0x7f22cc009b30).stop 2023-11-30T21:53:26.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.070+0000 7f22dcc0c700 1 -- 172.21.15.49:0/741015547 shutdown_connections 2023-11-30T21:53:26.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.070+0000 7f22dcc0c700 1 --2- 172.21.15.49:0/741015547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22d80f62a0 0x7f22d80f66c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:26.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.070+0000 7f22dcc0c700 1 -- 172.21.15.49:0/741015547 >> 172.21.15.49:0/741015547 conn(0x7f22d80f1aa0 msgr2=0x7f22d80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:26.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.070+0000 7f22dcc0c700 1 -- 172.21.15.49:0/741015547 shutdown_connections 2023-11-30T21:53:26.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.070+0000 7f22dcc0c700 1 -- 172.21.15.49:0/741015547 wait complete. 2023-11-30T21:53:26.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.070+0000 7f22dcc0c700 1 Processor -- start 2023-11-30T21:53:26.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.070+0000 7f22dcc0c700 1 -- start start 2023-11-30T21:53:26.073 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.070+0000 7f22dcc0c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22d80f62a0 0x7f22d806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:26.074 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.070+0000 7f22dcc0c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d8069e20 con 0x7f22d80f62a0 2023-11-30T21:53:26.074 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.071+0000 7f22d77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22d80f62a0 0x7f22d806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:26.074 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.071+0000 7f22d77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22d80f62a0 0x7f22d806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:50582/0 (socket says 172.21.15.49:50582) 2023-11-30T21:53:26.074 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.071+0000 7f22d77fe700 1 -- 172.21.15.49:0/2833807837 learned_addr learned my addr 172.21.15.49:0/2833807837 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:53:26.074 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.071+0000 7f22d77fe700 1 -- 172.21.15.49:0/2833807837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22cc005040 con 0x7f22d80f62a0 2023-11-30T21:53:26.075 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.071+0000 7f22d77fe700 1 --2- 172.21.15.49:0/2833807837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22d80f62a0 0x7f22d806b9a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f22cc013510 tx=0x7f22cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:26.075 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.071+0000 7f22bffff700 1 -- 172.21.15.49:0/2833807837 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22cc009320 con 0x7f22d80f62a0 2023-11-30T21:53:26.075 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.072+0000 7f22dcc0c700 1 -- 172.21.15.49:0/2833807837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d8069fc0 con 0x7f22d80f62a0 2023-11-30T21:53:26.075 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.072+0000 7f22dcc0c700 1 -- 172.21.15.49:0/2833807837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d806a450 con 0x7f22d80f62a0 2023-11-30T21:53:26.076 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.072+0000 7f22bffff700 1 -- 172.21.15.49:0/2833807837 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f22cc013f00 con 0x7f22d80f62a0 2023-11-30T21:53:26.076 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.072+0000 7f22bffff700 1 -- 172.21.15.49:0/2833807837 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22cc014ce0 con 0x7f22d80f62a0 2023-11-30T21:53:26.076 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.072+0000 7f22bffff700 1 -- 172.21.15.49:0/2833807837 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22cc0263a0 con 0x7f22d80f62a0 2023-11-30T21:53:26.076 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.072+0000 7f22dcc0c700 1 -- 172.21.15.49:0/2833807837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b8004fa0 con 0x7f22d80f62a0 2023-11-30T21:53:26.076 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.073+0000 7f22bffff700 1 --2- 172.21.15.49:0/2833807837 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f22c002ff60 0x7f22c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:26.076 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.073+0000 7f22bffff700 1 -- 172.21.15.49:0/2833807837 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f22cc040f20 con 0x7f22d80f62a0 2023-11-30T21:53:26.076 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.073+0000 7f22d6ffd700 1 --2- 172.21.15.49:0/2833807837 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f22c002ff60 0x7f22c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:26.076 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.073+0000 7f22d6ffd700 1 --2- 172.21.15.49:0/2833807837 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f22c002ff60 0x7f22c0032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f22c80096a0 tx=0x7f22c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:26.079 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.078+0000 7f22bffff700 1 -- 172.21.15.49:0/2833807837 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f22cc06d2b0 con 0x7f22d80f62a0 2023-11-30T21:53:26.301 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.299+0000 7f22dcc0c700 1 -- 172.21.15.49:0/2833807837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22b8005670 con 0x7f22d80f62a0 2023-11-30T21:53:26.302 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.300+0000 7f22bffff700 1 -- 172.21.15.49:0/2833807837 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f22cc0095f0 con 0x7f22d80f62a0 2023-11-30T21:53:26.302 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:53:26.302 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:53:26.303 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.301+0000 7f22dcc0c700 1 -- 172.21.15.49:0/2833807837 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f22c002ff60 msgr2=0x7f22c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:26.303 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.302+0000 7f22dcc0c700 1 --2- 172.21.15.49:0/2833807837 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f22c002ff60 0x7f22c0032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f22c80096a0 tx=0x7f22c8006b40).stop 2023-11-30T21:53:26.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.302+0000 7f22dcc0c700 1 -- 172.21.15.49:0/2833807837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22d80f62a0 msgr2=0x7f22d806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:26.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.302+0000 7f22dcc0c700 1 --2- 172.21.15.49:0/2833807837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22d80f62a0 0x7f22d806b9a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f22cc013510 tx=0x7f22cc00b040).stop 2023-11-30T21:53:26.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.302+0000 7f22dcc0c700 1 -- 172.21.15.49:0/2833807837 shutdown_connections 2023-11-30T21:53:26.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.302+0000 7f22dcc0c700 1 --2- 172.21.15.49:0/2833807837 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f22c002ff60 0x7f22c0032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:26.304 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.302+0000 7f22dcc0c700 1 --2- 172.21.15.49:0/2833807837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22d80f62a0 0x7f22d806b9a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:26.305 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.302+0000 7f22dcc0c700 1 -- 172.21.15.49:0/2833807837 >> 172.21.15.49:0/2833807837 conn(0x7f22d80f1aa0 msgr2=0x7f22d80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:26.305 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.302+0000 7f22dcc0c700 1 -- 172.21.15.49:0/2833807837 shutdown_connections 2023-11-30T21:53:26.305 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:26.302+0000 7f22dcc0c700 1 -- 172.21.15.49:0/2833807837 wait complete. 2023-11-30T21:53:26.305 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:53:27.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:27 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:25.548464+0000 mgr.smithi027.brrjcl (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:27.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:27 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:53:26.301672+0000 mon.smithi027 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.49:0/2833807837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:53:27.748 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:53:27.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:53:29.006 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.004+0000 7f759ea83700 1 -- 172.21.15.49:0/1607999683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75980f5410 msgr2=0x7f75980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:29.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.004+0000 7f759ea83700 1 --2- 172.21.15.49:0/1607999683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75980f5410 0x7f75980f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f758c009fb0 tx=0x7f758c009b30).stop 2023-11-30T21:53:29.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.004+0000 7f759ea83700 1 -- 172.21.15.49:0/1607999683 shutdown_connections 2023-11-30T21:53:29.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.004+0000 7f759ea83700 1 --2- 172.21.15.49:0/1607999683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75980f5410 0x7f75980f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:29.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.004+0000 7f759ea83700 1 -- 172.21.15.49:0/1607999683 >> 172.21.15.49:0/1607999683 conn(0x7f75980f0b70 msgr2=0x7f75980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:29.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.004+0000 7f759ea83700 1 -- 172.21.15.49:0/1607999683 shutdown_connections 2023-11-30T21:53:29.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.004+0000 7f759ea83700 1 -- 172.21.15.49:0/1607999683 wait complete. 2023-11-30T21:53:29.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.004+0000 7f759ea83700 1 Processor -- start 2023-11-30T21:53:29.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.005+0000 7f759ea83700 1 -- start start 2023-11-30T21:53:29.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.005+0000 7f759ea83700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75980f5410 0x7f75980fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:29.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.005+0000 7f759ea83700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75980fb820 con 0x7f75980f5410 2023-11-30T21:53:29.007 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.005+0000 7f759c81f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75980f5410 0x7f75980fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:29.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.005+0000 7f759c81f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75980f5410 0x7f75980fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:54290/0 (socket says 172.21.15.49:54290) 2023-11-30T21:53:29.008 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.005+0000 7f759c81f700 1 -- 172.21.15.49:0/491014548 learned_addr learned my addr 172.21.15.49:0/491014548 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:53:29.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.005+0000 7f759c81f700 1 -- 172.21.15.49:0/491014548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758c005040 con 0x7f75980f5410 2023-11-30T21:53:29.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.006+0000 7f759c81f700 1 --2- 172.21.15.49:0/491014548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75980f5410 0x7f75980fd3a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f758c013510 tx=0x7f758c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:29.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.006+0000 7f75957fa700 1 -- 172.21.15.49:0/491014548 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f758c009320 con 0x7f75980f5410 2023-11-30T21:53:29.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.006+0000 7f759ea83700 1 -- 172.21.15.49:0/491014548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75980fb9c0 con 0x7f75980f5410 2023-11-30T21:53:29.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.006+0000 7f75957fa700 1 -- 172.21.15.49:0/491014548 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f758c013f00 con 0x7f75980f5410 2023-11-30T21:53:29.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.006+0000 7f75957fa700 1 -- 172.21.15.49:0/491014548 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f758c014d90 con 0x7f75980f5410 2023-11-30T21:53:29.009 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.006+0000 7f759ea83700 1 -- 172.21.15.49:0/491014548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75980fbe50 con 0x7f75980f5410 2023-11-30T21:53:29.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.007+0000 7f75957fa700 1 -- 172.21.15.49:0/491014548 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f758c0263a0 con 0x7f75980f5410 2023-11-30T21:53:29.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.007+0000 7f75957fa700 1 --2- 172.21.15.49:0/491014548 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f758802ff60 0x7f7588032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:29.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.007+0000 7f75957fa700 1 -- 172.21.15.49:0/491014548 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f758c040ff0 con 0x7f75980f5410 2023-11-30T21:53:29.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.007+0000 7f759ea83700 1 -- 172.21.15.49:0/491014548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7598059100 con 0x7f75980f5410 2023-11-30T21:53:29.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.007+0000 7f7597fff700 1 --2- 172.21.15.49:0/491014548 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f758802ff60 0x7f7588032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:29.010 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.007+0000 7f7597fff700 1 --2- 172.21.15.49:0/491014548 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f758802ff60 0x7f7588032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f75840096a0 tx=0x7f7584006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:29.013 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.012+0000 7f75957fa700 1 -- 172.21.15.49:0/491014548 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f758c012920 con 0x7f75980f5410 2023-11-30T21:53:29.229 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.228+0000 7f759ea83700 1 -- 172.21.15.49:0/491014548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75980fc550 con 0x7f75980f5410 2023-11-30T21:53:29.230 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.228+0000 7f75957fa700 1 -- 172.21.15.49:0/491014548 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f758c02b020 con 0x7f75980f5410 2023-11-30T21:53:29.230 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:53:29.230 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:53:29.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.230+0000 7f759ea83700 1 -- 172.21.15.49:0/491014548 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f758802ff60 msgr2=0x7f7588032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:29.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.230+0000 7f759ea83700 1 --2- 172.21.15.49:0/491014548 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f758802ff60 0x7f7588032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f75840096a0 tx=0x7f7584006b40).stop 2023-11-30T21:53:29.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.230+0000 7f759ea83700 1 -- 172.21.15.49:0/491014548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75980f5410 msgr2=0x7f75980fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:29.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.230+0000 7f759ea83700 1 --2- 172.21.15.49:0/491014548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75980f5410 0x7f75980fd3a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f758c013510 tx=0x7f758c00b040).stop 2023-11-30T21:53:29.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.230+0000 7f759ea83700 1 -- 172.21.15.49:0/491014548 shutdown_connections 2023-11-30T21:53:29.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.230+0000 7f759ea83700 1 --2- 172.21.15.49:0/491014548 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f758802ff60 0x7f7588032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:29.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.230+0000 7f759ea83700 1 --2- 172.21.15.49:0/491014548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75980f5410 0x7f75980fd3a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:29.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.230+0000 7f759ea83700 1 -- 172.21.15.49:0/491014548 >> 172.21.15.49:0/491014548 conn(0x7f75980f0b70 msgr2=0x7f759805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:29.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.230+0000 7f759ea83700 1 -- 172.21.15.49:0/491014548 shutdown_connections 2023-11-30T21:53:29.232 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:29.230+0000 7f759ea83700 1 -- 172.21.15.49:0/491014548 wait complete. 2023-11-30T21:53:29.233 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:53:29.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:27.548788+0000 mgr.smithi027.brrjcl (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:29.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:29 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:53:29.230019+0000 mon.smithi027 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.49:0/491014548' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:53:30.604 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-30T21:53:30.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-30T21:53:31.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:31 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: cluster 2023-11-30T21:53:29.549103+0000 mgr.smithi027.brrjcl (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-30T21:53:31.974 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.971+0000 7f2572bc4700 1 -- 172.21.15.49:0/2702554851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256c0f5410 msgr2=0x7f256c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:31.974 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.971+0000 7f2572bc4700 1 --2- 172.21.15.49:0/2702554851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256c0f5410 0x7f256c0f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f2560009fb0 tx=0x7f2560009b30).stop 2023-11-30T21:53:31.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.971+0000 7f2572bc4700 1 -- 172.21.15.49:0/2702554851 shutdown_connections 2023-11-30T21:53:31.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.971+0000 7f2572bc4700 1 --2- 172.21.15.49:0/2702554851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256c0f5410 0x7f256c0f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:31.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.971+0000 7f2572bc4700 1 -- 172.21.15.49:0/2702554851 >> 172.21.15.49:0/2702554851 conn(0x7f256c0f0b70 msgr2=0x7f256c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:31.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.972+0000 7f2572bc4700 1 -- 172.21.15.49:0/2702554851 shutdown_connections 2023-11-30T21:53:31.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.972+0000 7f2572bc4700 1 -- 172.21.15.49:0/2702554851 wait complete. 2023-11-30T21:53:31.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.972+0000 7f2572bc4700 1 Processor -- start 2023-11-30T21:53:31.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.972+0000 7f2572bc4700 1 -- start start 2023-11-30T21:53:31.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.972+0000 7f2572bc4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256c0f5410 0x7f256c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:31.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.972+0000 7f2572bc4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c0fb820 con 0x7f256c0f5410 2023-11-30T21:53:31.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.973+0000 7f2570960700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256c0f5410 0x7f256c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:31.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.973+0000 7f2570960700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256c0f5410 0x7f256c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.49:54304/0 (socket says 172.21.15.49:54304) 2023-11-30T21:53:31.975 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.973+0000 7f2570960700 1 -- 172.21.15.49:0/2923005506 learned_addr learned my addr 172.21.15.49:0/2923005506 (peer_addr_for_me v2:172.21.15.49:0/0) 2023-11-30T21:53:31.976 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.973+0000 7f2570960700 1 -- 172.21.15.49:0/2923005506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2560005040 con 0x7f256c0f5410 2023-11-30T21:53:31.976 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.973+0000 7f2570960700 1 --2- 172.21.15.49:0/2923005506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256c0f5410 0x7f256c0fd3a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f2560013510 tx=0x7f256000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:31.976 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.973+0000 7f25697fa700 1 -- 172.21.15.49:0/2923005506 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2560009320 con 0x7f256c0f5410 2023-11-30T21:53:31.976 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.973+0000 7f2572bc4700 1 -- 172.21.15.49:0/2923005506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f256c0fb9c0 con 0x7f256c0f5410 2023-11-30T21:53:31.976 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.974+0000 7f25697fa700 1 -- 172.21.15.49:0/2923005506 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2560013f00 con 0x7f256c0f5410 2023-11-30T21:53:31.976 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.974+0000 7f25697fa700 1 -- 172.21.15.49:0/2923005506 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2560014d90 con 0x7f256c0f5410 2023-11-30T21:53:31.976 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.974+0000 7f2572bc4700 1 -- 172.21.15.49:0/2923005506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256c0fbe50 con 0x7f256c0f5410 2023-11-30T21:53:31.977 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.974+0000 7f25697fa700 1 -- 172.21.15.49:0/2923005506 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25600095f0 con 0x7f256c0f5410 2023-11-30T21:53:31.977 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.974+0000 7f2572bc4700 1 -- 172.21.15.49:0/2923005506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256c059100 con 0x7f256c0f5410 2023-11-30T21:53:31.977 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.974+0000 7f25697fa700 1 --2- 172.21.15.49:0/2923005506 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f255c02ff60 0x7f255c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-30T21:53:31.977 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.975+0000 7f25697fa700 1 -- 172.21.15.49:0/2923005506 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25600410c0 con 0x7f256c0f5410 2023-11-30T21:53:31.977 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.975+0000 7f256bfff700 1 --2- 172.21.15.49:0/2923005506 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f255c02ff60 0x7f255c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-30T21:53:31.977 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.975+0000 7f256bfff700 1 --2- 172.21.15.49:0/2923005506 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f255c02ff60 0x7f255c032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f25580096a0 tx=0x7f2558006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-30T21:53:31.980 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:31.979+0000 7f25697fa700 1 -- 172.21.15.49:0/2923005506 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2560014420 con 0x7f256c0f5410 2023-11-30T21:53:32.198 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:32.197+0000 7f2572bc4700 1 -- 172.21.15.49:0/2923005506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f256c0fae60 con 0x7f256c0f5410 2023-11-30T21:53:32.198 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:32.198+0000 7f25697fa700 1 -- 172.21.15.49:0/2923005506 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2560010230 con 0x7f256c0f5410 2023-11-30T21:53:32.199 INFO:teuthology.orchestra.run.smithi049.stdout: 2023-11-30T21:53:32.199 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"fsid":"6473e080-8fc9-11ee-95a2-87774f69a715","modified":"2023-11-30T21:43:08.685486Z","created":"2023-11-30T21:43:08.685486Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-30T21:53:32.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:32.199+0000 7f2572bc4700 1 -- 172.21.15.49:0/2923005506 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f255c02ff60 msgr2=0x7f255c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:32.200 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:32.199+0000 7f2572bc4700 1 --2- 172.21.15.49:0/2923005506 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f255c02ff60 0x7f255c032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f25580096a0 tx=0x7f2558006b40).stop 2023-11-30T21:53:32.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:32.199+0000 7f2572bc4700 1 -- 172.21.15.49:0/2923005506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256c0f5410 msgr2=0x7f256c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-30T21:53:32.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:32.199+0000 7f2572bc4700 1 --2- 172.21.15.49:0/2923005506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256c0f5410 0x7f256c0fd3a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f2560013510 tx=0x7f256000b040).stop 2023-11-30T21:53:32.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:32.200+0000 7f2572bc4700 1 -- 172.21.15.49:0/2923005506 shutdown_connections 2023-11-30T21:53:32.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:32.200+0000 7f2572bc4700 1 --2- 172.21.15.49:0/2923005506 >> [v2:172.21.15.27:6800/4041101911,v1:172.21.15.27:6801/4041101911] conn(0x7f255c02ff60 0x7f255c032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:32.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:32.200+0000 7f2572bc4700 1 --2- 172.21.15.49:0/2923005506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256c0f5410 0x7f256c0fd3a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-30T21:53:32.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:32.200+0000 7f2572bc4700 1 -- 172.21.15.49:0/2923005506 >> 172.21.15.49:0/2923005506 conn(0x7f256c0f0b70 msgr2=0x7f256c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-30T21:53:32.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:32.200+0000 7f2572bc4700 1 -- 172.21.15.49:0/2923005506 shutdown_connections 2023-11-30T21:53:32.201 INFO:teuthology.orchestra.run.smithi049.stderr:2023-11-30T21:53:32.200+0000 7f2572bc4700 1 -- 172.21.15.49:0/2923005506 wait complete. 2023-11-30T21:53:32.202 INFO:teuthology.orchestra.run.smithi049.stderr:dumped monmap epoch 1 2023-11-30T21:53:32.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:32 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: audit 2023-11-30T21:53:32.198369+0000 mon.smithi027 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.49:0/2923005506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-30T21:53:32.643 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-11-30T21:53:32.646 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-30T21:53:32.646 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-30T21:53:32.672 DEBUG:teuthology.orchestra.run.smithi049:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-30T21:53:32.691 INFO:tasks.cephadm:Stopping all daemons... 2023-11-30T21:53:32.691 INFO:tasks.cephadm.mon.smithi027:Stopping mon.smithi027... 2023-11-30T21:53:32.691 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi027 2023-11-30T21:53:33.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:32 smithi027 systemd[1]: Stopping Ceph mon.smithi027 for 6473e080-8fc9-11ee-95a2-87774f69a715... 2023-11-30T21:53:33.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:32 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:53:32.817+0000 7fab67fef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi027 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-30T21:53:33.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Nov 30 21:53:32 smithi027 ceph-6473e080-8fc9-11ee-95a2-87774f69a715-mon.smithi027[140617]: debug 2023-11-30T21:53:32.817+0000 7fab67fef700 -1 mon.smithi027@0(leader) e1 *** Got Signal Terminated *** 2023-11-30T21:53:33.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi027.service' 2023-11-30T21:53:33.227 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-30T21:53:33.227 INFO:tasks.cephadm.mon.smithi027:Stopped mon.smithi027 2023-11-30T21:53:33.227 INFO:tasks.cephadm.mon.smithi049:Stopping mon.smithi049... 2023-11-30T21:53:33.228 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi049 2023-11-30T21:53:33.263 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi049.service: Unit ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi049.service not loaded. 2023-11-30T21:53:33.264 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-11-30T21:53:33.265 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi049" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 5: 'sudo systemctl stop ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi049' 2023-11-30T21:53:33.266 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-30T21:53:33.268 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472522/remote/ubuntu@smithi027.front.sepia.ceph.com/crash 2023-11-30T21:53:33.269 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/crash -- . 2023-11-30T21:53:33.314 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472522/remote/ubuntu@smithi049.front.sepia.ceph.com/crash 2023-11-30T21:53:33.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/crash -- . 2023-11-30T21:53:33.360 INFO:teuthology.orchestra.run.smithi049.stderr:tar: /var/lib/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/crash: Cannot open: No such file or directory 2023-11-30T21:53:33.360 INFO:teuthology.orchestra.run.smithi049.stderr:tar: Error is not recoverable: exiting now 2023-11-30T21:53:33.361 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-30T21:53:33.361 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-30T21:53:33.392 INFO:teuthology.orchestra.run.smithi027.stderr:grep: /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/ceph.log: No such file or directory 2023-11-30T21:53:33.394 INFO:tasks.cephadm:Compressing logs... 2023-11-30T21:53:33.395 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-30T21:53:33.437 DEBUG:teuthology.orchestra.run.smithi049:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-30T21:53:33.463 INFO:teuthology.orchestra.run.smithi027.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-30T21:53:33.463 INFO:teuthology.orchestra.run.smithi027.stderr:: No such file or directory 2023-11-30T21:53:33.464 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/ceph-volume.log 2023-11-30T21:53:33.464 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/ceph-mgr.smithi027.brrjcl.log 2023-11-30T21:53:33.464 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/ceph-mon.smithi027.log 2023-11-30T21:53:33.464 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/ceph-volume.log: /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/ceph-mgr.smithi027.brrjcl.log: /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/ceph-mon.smithi027.log: 88.3% -- replaced with /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/ceph-volume.log.gz 2023-11-30T21:53:33.466 INFO:teuthology.orchestra.run.smithi049.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-11-30T21:53:33.466 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-30T21:53:33.468 INFO:teuthology.orchestra.run.smithi027.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-30T21:53:33.468 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/cephadm.log: 96.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-30T21:53:33.470 INFO:teuthology.orchestra.run.smithi049.stderr: 2023-11-30T21:53:33.470 INFO:teuthology.orchestra.run.smithi049.stderr:real 0m0.014s 2023-11-30T21:53:33.470 INFO:teuthology.orchestra.run.smithi049.stderr:user 0m0.010s 2023-11-30T21:53:33.470 INFO:teuthology.orchestra.run.smithi049.stderr:sys 0m0.013s 2023-11-30T21:53:33.545 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/ceph-mgr.smithi027.brrjcl.log: file size changed while zipping 2023-11-30T21:53:33.545 INFO:teuthology.orchestra.run.smithi027.stderr: 92.7% -- replaced with /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/ceph-mgr.smithi027.brrjcl.log.gz 2023-11-30T21:53:33.616 INFO:teuthology.orchestra.run.smithi027.stderr: 90.6% -- replaced with /var/log/ceph/6473e080-8fc9-11ee-95a2-87774f69a715/ceph-mon.smithi027.log.gz 2023-11-30T21:53:33.618 INFO:teuthology.orchestra.run.smithi027.stderr: 2023-11-30T21:53:33.618 INFO:teuthology.orchestra.run.smithi027.stderr:real 0m0.166s 2023-11-30T21:53:33.618 INFO:teuthology.orchestra.run.smithi027.stderr:user 0m0.238s 2023-11-30T21:53:33.618 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m0.024s 2023-11-30T21:53:33.619 INFO:tasks.cephadm:Archiving logs... 2023-11-30T21:53:33.619 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472522/remote/ubuntu@smithi027.front.sepia.ceph.com/log 2023-11-30T21:53:33.620 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-30T21:53:33.730 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472522/remote/ubuntu@smithi049.front.sepia.ceph.com/log 2023-11-30T21:53:33.731 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-30T21:53:33.763 INFO:tasks.cephadm:Removing cluster... 2023-11-30T21:53:33.763 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 --force 2023-11-30T21:54:03.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6473e080-8fc9-11ee-95a2-87774f69a715 --force 2023-11-30T21:54:04.047 INFO:tasks.cephadm:Removing cephadm ... 2023-11-30T21:54:04.047 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-30T21:54:04.069 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-30T21:54:04.092 INFO:tasks.cephadm:Teardown complete 2023-11-30T21:54:04.092 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 5: 'sudo systemctl stop ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi049' 2023-11-30T21:54:04.173 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5a4d7365fc614cb0bd7527996997eb9c Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0dc074ce0b0b3b4f393f56dd32ef36884ebf45a9/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_595fad1080bd379e3c0231a8e02595924c59a9da/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi049 with status 5: 'sudo systemctl stop ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi049' 2023-11-30T21:54:04.180 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-30T21:54:04.210 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-30T21:54:04.262 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-30T21:54:04.262 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-30T21:54:04.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-30T21:54:04.356 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-30T21:54:04.366 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-30T21:54:04.366 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-30T21:54:04.368 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-30T21:54:04.384 INFO:teuthology.orchestra.run.smithi027.stderr:bash: ntpq: command not found 2023-11-30T21:54:04.386 INFO:teuthology.orchestra.run.smithi049.stderr:bash: ntpq: command not found 2023-11-30T21:54:04.388 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-30T21:54:04.389 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2023-11-30T21:54:04.389 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 +1295us[+1295us] +/- 81ms 2023-11-30T21:54:04.389 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -553us[ -521us] +/- 76ms 2023-11-30T21:54:04.389 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -563us[ -531us] +/- 66ms 2023-11-30T21:54:04.389 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-30T21:54:04.390 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-30T21:54:04.390 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2023-11-30T21:54:04.390 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 +1271us[+1271us] +/- 81ms 2023-11-30T21:54:04.390 INFO:teuthology.orchestra.run.smithi049.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -567us[ -574us] +/- 76ms 2023-11-30T21:54:04.390 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 -554us[ -562us] +/- 66ms 2023-11-30T21:54:04.390 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-30T21:54:04.392 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-30T21:54:04.403 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-30T21:54:04.404 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-30T21:54:04.444 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-30T21:54:04.538 DEBUG:teuthology.orchestra.run.smithi049:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-30T21:54:04.634 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-30T21:54:04.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-30T21:54:04.668 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-30T21:54:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-30T21:54:04.701 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-30T21:54:04.710 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-30T21:54:04.776 INFO:teuthology.task.internal:Duration was 1389.492884 seconds 2023-11-30T21:54:04.776 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-30T21:54:04.863 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-30T21:54:04.864 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-30T21:54:04.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-30T21:54:04.913 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-30T21:54:04.914 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-30T21:54:05.379 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-30T21:54:05.379 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2023-11-30T21:54:05.380 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-30T21:54:05.405 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi049.front.sepia.ceph.com 2023-11-30T21:54:05.405 DEBUG:teuthology.orchestra.run.smithi049:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-30T21:54:05.430 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-30T21:54:05.430 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-30T21:54:05.447 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-30T21:54:05.519 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-30T21:54:05.519 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-30T21:54:05.521 DEBUG:teuthology.orchestra.run.smithi049:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-30T21:54:05.621 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-30T21:54:05.632 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-30T21:54:05.632 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-30T21:54:05.660 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-30T21:54:05.688 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-30T21:54:05.708 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-30T21:54:05.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-30T21:54:05.735 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2023-11-30T21:54:05.755 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = core 2023-11-30T21:54:05.775 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-30T21:54:05.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-30T21:54:05.816 DEBUG:teuthology.orchestra.run.smithi049:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-30T21:54:05.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-30T21:54:05.834 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-30T21:54:05.845 INFO:teuthology.task.internal:Transferring archived files... 2023-11-30T21:54:05.846 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472522/remote/smithi027 2023-11-30T21:54:05.847 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-30T21:54:05.897 DEBUG:teuthology.misc:Transferring archived files from smithi049:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472522/remote/smithi049 2023-11-30T21:54:05.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-30T21:54:05.937 INFO:teuthology.task.internal:Removing archive directory... 2023-11-30T21:54:05.938 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-30T21:54:05.940 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-30T21:54:05.988 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-30T21:54:05.999 INFO:teuthology.task.internal:Not uploading archives. 2023-11-30T21:54:05.999 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-30T21:54:06.039 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-30T21:54:06.040 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-30T21:54:06.042 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-30T21:54:06.058 INFO:teuthology.orchestra.run.smithi027.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 30 21:54 /home/ubuntu/cephtest 2023-11-30T21:54:06.060 INFO:teuthology.orchestra.run.smithi049.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 30 21:54 /home/ubuntu/cephtest 2023-11-30T21:54:06.061 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-30T21:54:06.082 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-30T21:54:06.199 INFO:teuthology.nuke:Checking targets against current locks 2023-11-30T21:54:06.224 DEBUG:teuthology.nuke:shortname: smithi027 2023-11-30T21:54:06.224 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-30T21:54:06.249 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472522', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-30 21:08:06.758714', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKKRK2atzkSY9j+qZTWr7qs/a/nYUKdIusUMpUbHzkWBMMbs9R5dv2bBqbn2OrOcr9rcbdFU5OI3O/Zr3l7/jTs='} 2023-11-30T21:54:06.253 DEBUG:teuthology.nuke:shortname: smithi049 2023-11-30T21:54:06.253 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-30T21:54:06.275 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-30_16:00:27-rados-wip-yuri8-testing-2023-11-29-0706-pacific-distro-default-smithi/7472522', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-30 21:08:06.760380', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDxM9L1/quSHWDQf/7twLLjDuknYBeaRc7dwxOgYHbExgrqE1RlX6dkiwi3ClkqBEa8Gve7LPj3Gv5Sv63MgfpA='} 2023-11-30T21:54:06.312 INFO:teuthology.orchestra.console.smithi027:Power off 2023-11-30T21:54:06.312 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-30T21:54:06.341 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off 2023-11-30T21:54:06.343 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-30T21:54:06.364 INFO:teuthology.orchestra.console.smithi049:Power off 2023-11-30T21:54:06.364 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-30T21:54:06.379 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2023-11-30T21:54:06.393 DEBUG:teuthology.orchestra.console.smithi049:power off output: Chassis Power Control: Down/Off 2023-11-30T21:54:06.393 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-30T21:54:06.422 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2023-11-30T21:54:10.380 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-30T21:54:10.425 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-30T21:54:10.491 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2023-11-30T21:54:10.536 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2023-11-30T21:54:14.494 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-30T21:54:14.537 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-30T21:54:14.607 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2023-11-30T21:54:14.647 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is off 2023-11-30T21:54:14.748 INFO:teuthology.orchestra.console.smithi049:Power off completed 2023-11-30T21:54:18.610 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-30T21:54:18.720 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off 2023-11-30T21:54:18.821 INFO:teuthology.orchestra.console.smithi027:Power off completed 2023-11-30T21:54:18.929 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1389.4928843975067 failure_reason: 'Command failed on smithi049 with status 5: ''sudo systemctl stop ceph-6473e080-8fc9-11ee-95a2-87774f69a715@mon.smithi049''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5a4d7365fc614cb0bd7527996997eb9c status: fail success: false 2023-11-30T21:54:18.930 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-30T21:54:19.018 INFO:teuthology.run:FAIL