2024-02-19T21:54:18.405 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-19T21:54:18.405 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566805 -- /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566805/orig.config.yaml 2024-02-19T21:54:18.455 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T21:54:18.539 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566805 branch: pacific-release description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7566805' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: pacific-release ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(PG_DEGRADED\) - Degraded data redundancy - \(PG_ - MDS_INSUFFICIENT_STANDBY - deprecated feature inline_data - compat changed unexpectedly - overall HEALTH_ - OSD_DOWN - OSD_ - but it is still running - is not responding sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: pacific-release sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 9538 sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: pacific-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 targets: smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPf///913xOrL2NMpXOzOI/M8zYvLGCOT37fQKf5ikaK68NoO9PiHo6l+sRoZmZeWo5+OdjegIenv9V9zakyIkg= smithi184.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNiXeisfqaKrCGXPH/j/m8EDYZT0uEObYnnTxCi4UKUJ1XvQI2qhHoJCoEJkLXSVTJW8IMDsRnpa9A1qDfh7wgg= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_19:27:49 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-19T21:54:18.539 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa; will attempt to use it 2024-02-19T21:54:18.540 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks 2024-02-19T21:54:18.540 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-19T21:54:18.542 INFO:teuthology.task.internal:Checking packages... 2024-02-19T21:54:18.565 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '81bd20d634209c7cb82c18be12b4b5a05643ebf1' 2024-02-19T21:54:18.565 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T21:54:18.565 INFO:teuthology.packaging:ref: None 2024-02-19T21:54:18.565 INFO:teuthology.packaging:tag: None 2024-02-19T21:54:18.565 INFO:teuthology.packaging:branch: pacific-release 2024-02-19T21:54:18.566 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:54:18.566 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific-release 2024-02-19T21:54:18.787 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-548.g81bd20d6 2024-02-19T21:54:18.788 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-19T21:54:18.794 INFO:teuthology.task.internal:no buildpackages task found 2024-02-19T21:54:18.794 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-19T21:54:18.852 INFO:teuthology.task.internal:Saving configuration 2024-02-19T21:54:18.880 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-19T21:54:18.885 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T21:54:18.909 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566805', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:49:15.896682', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPf///913xOrL2NMpXOzOI/M8zYvLGCOT37fQKf5ikaK68NoO9PiHo6l+sRoZmZeWo5+OdjegIenv9V9zakyIkg='} 2024-02-19T21:54:18.932 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566805', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:49:15.898544', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNiXeisfqaKrCGXPH/j/m8EDYZT0uEObYnnTxCi4UKUJ1XvQI2qhHoJCoEJkLXSVTJW8IMDsRnpa9A1qDfh7wgg='} 2024-02-19T21:54:18.933 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-19T21:54:18.938 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-19T21:54:18.939 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-19T21:54:18.939 INFO:teuthology.run_tasks:Running task console_log... 2024-02-19T21:54:19.034 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdc19b42f70>, signals=[15]) 2024-02-19T21:54:19.034 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-19T21:54:19.040 INFO:teuthology.task.internal:Opening connections... 2024-02-19T21:54:19.040 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-02-19T21:54:19.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:54:19.120 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2024-02-19T21:54:19.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:54:19.193 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-19T21:54:19.199 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-02-19T21:54:19.222 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-02-19T21:54:19.222 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-02-19T21:54:19.282 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2024-02-19T21:54:19.283 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="8" 2024-02-19T21:54:19.283 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2024-02-19T21:54:19.283 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2024-02-19T21:54:19.283 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="8" 2024-02-19T21:54:19.283 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el8" 2024-02-19T21:54:19.283 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T21:54:19.283 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-02-19T21:54:19.283 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T21:54:19.283 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2024-02-19T21:54:19.283 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T21:54:19.283 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T21:54:19.283 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T21:54:19.284 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-02-19T21:54:19.303 DEBUG:teuthology.orchestra.run.smithi184:> uname -m 2024-02-19T21:54:19.326 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2024-02-19T21:54:19.326 DEBUG:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2024-02-19T21:54:19.388 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="CentOS Stream" 2024-02-19T21:54:19.388 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="8" 2024-02-19T21:54:19.388 INFO:teuthology.orchestra.run.smithi184.stdout:ID="centos" 2024-02-19T21:54:19.388 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="rhel fedora" 2024-02-19T21:54:19.388 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="8" 2024-02-19T21:54:19.388 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el8" 2024-02-19T21:54:19.389 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T21:54:19.389 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2024-02-19T21:54:19.389 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T21:54:19.389 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://centos.org/" 2024-02-19T21:54:19.389 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T21:54:19.389 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T21:54:19.389 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T21:54:19.389 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2024-02-19T21:54:19.412 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-19T21:54:19.423 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-19T21:54:19.460 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-19T21:54:19.461 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-02-19T21:54:19.463 DEBUG:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2024-02-19T21:54:19.484 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-19T21:54:19.494 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-19T21:54:19.494 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-02-19T21:54:19.524 DEBUG:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2024-02-19T21:54:19.553 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-19T21:54:19.598 INFO:teuthology.run_tasks:Running task kernel... 2024-02-19T21:54:19.613 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-19T21:54:19.613 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-19T21:54:19.613 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T21:54:19.614 DEBUG:teuthology.orchestra.run.smithi184:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T21:54:19.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:54:19.635 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-02-19T21:54:19.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:54:19.636 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-02-19T21:54:19.695 INFO:teuthology.orchestra.run.smithi176.stdout:4.18.0-489.el8.x86_64 2024-02-19T21:54:19.696 INFO:teuthology.task.kernel:Running kernel on smithi176: 4.18.0-489.el8.x86_64 2024-02-19T21:54:19.696 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum install -y kernel 2024-02-19T21:54:19.698 INFO:teuthology.orchestra.run.smithi184.stdout:4.18.0-489.el8.x86_64 2024-02-19T21:54:19.698 INFO:teuthology.task.kernel:Running kernel on smithi184: 4.18.0-489.el8.x86_64 2024-02-19T21:54:19.698 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum install -y kernel 2024-02-19T21:54:32.613 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - AppStream 2.6 MB/s | 27 MB 00:10 2024-02-19T21:54:33.035 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 8 - AppStream 2.5 MB/s | 27 MB 00:10 2024-02-19T21:54:38.013 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - BaseOS 18 MB/s | 10 MB 00:00 2024-02-19T21:54:38.569 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 10 MB 00:00 2024-02-19T21:54:39.756 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - Extras 136 kB/s | 18 kB 00:00 2024-02-19T21:54:39.998 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - Extras common packages 52 kB/s | 6.9 kB 00:00 2024-02-19T21:54:40.311 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 8 - Extras 134 kB/s | 18 kB 00:00 2024-02-19T21:54:40.332 INFO:teuthology.orchestra.run.smithi176.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2024-02-19T21:54:40.545 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 8 - Extras common packages 56 kB/s | 6.9 kB 00:00 2024-02-19T21:54:40.772 INFO:teuthology.orchestra.run.smithi184.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-02-19T21:54:41.200 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2024-02-19T21:54:42.666 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 16 MB 00:01 2024-02-19T21:54:45.050 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 613 kB/s | 24 kB 00:00 2024-02-19T21:54:46.414 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 465 kB/s | 24 kB 00:00 2024-02-19T21:54:47.955 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T21:54:47.955 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T21:54:48.081 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T21:54:48.081 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T21:54:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-02-19T21:54:48.082 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T21:54:48.082 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T21:54:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T21:54:48.082 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-02-19T21:54:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T21:54:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T21:54:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T21:54:48.082 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T21:54:48.082 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T21:54:48.082 INFO:teuthology.orchestra.run.smithi176.stdout:Install 3 Packages 2024-02-19T21:54:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T21:54:48.083 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 90 M 2024-02-19T21:54:48.083 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 97 M 2024-02-19T21:54:48.083 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T21:54:49.234 INFO:teuthology.orchestra.run.smithi176.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 11 MB/s | 10 MB 00:00 2024-02-19T21:54:49.361 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T21:54:49.361 INFO:teuthology.orchestra.run.smithi184.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T21:54:49.468 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T21:54:49.469 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T21:54:49.469 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-02-19T21:54:49.469 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T21:54:49.469 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T21:54:49.469 INFO:teuthology.orchestra.run.smithi184.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T21:54:49.469 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-02-19T21:54:49.469 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T21:54:49.469 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T21:54:49.470 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T21:54:49.470 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T21:54:49.470 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T21:54:49.470 INFO:teuthology.orchestra.run.smithi184.stdout:Install 3 Packages 2024-02-19T21:54:49.470 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T21:54:49.470 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 90 M 2024-02-19T21:54:49.470 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 97 M 2024-02-19T21:54:49.471 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T21:54:50.757 INFO:teuthology.orchestra.run.smithi184.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 8.4 MB/s | 10 MB 00:01 2024-02-19T21:54:51.468 INFO:teuthology.orchestra.run.smithi176.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 11 MB/s | 36 MB 00:03 2024-02-19T21:54:52.078 INFO:teuthology.orchestra.run.smithi176.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 11 MB/s | 43 MB 00:03 2024-02-19T21:54:52.078 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:52.078 INFO:teuthology.orchestra.run.smithi176.stdout:Total 22 MB/s | 90 MB 00:03 2024-02-19T21:54:52.758 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T21:54:52.765 INFO:teuthology.orchestra.run.smithi184.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 11 MB/s | 36 MB 00:03 2024-02-19T21:54:53.179 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T21:54:53.179 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T21:54:53.424 INFO:teuthology.orchestra.run.smithi184.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 11 MB/s | 43 MB 00:03 2024-02-19T21:54:53.424 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:54:53.424 INFO:teuthology.orchestra.run.smithi184.stdout:Total 23 MB/s | 90 MB 00:03 2024-02-19T21:54:54.106 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T21:54:54.420 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T21:54:54.443 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T21:54:54.526 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T21:54:54.526 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T21:54:55.760 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T21:54:55.773 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T21:54:57.208 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T21:54:57.688 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:54:58.539 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T21:54:59.057 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:55:01.229 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:55:01.621 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:55:02.642 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:55:03.023 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:55:05.048 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:55:05.361 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:55:06.464 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:55:06.779 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:55:27.224 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:55:27.247 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:55:27.595 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:55:27.595 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:55:27.595 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:55:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:55:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T21:55:28.132 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T21:55:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T21:55:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T21:55:28.132 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T21:55:28.133 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T21:55:28.271 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:55:28.484 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:55:28.484 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:55:28.485 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T21:55:28.485 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T21:55:28.485 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T21:55:28.485 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-19T21:55:28.485 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T21:55:28.485 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:55:28.563 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:55:28.586 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:55:28.696 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:55:28.696 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:55:28.697 DEBUG:teuthology.orchestra.run.smithi176:> sudo rpm -qi grub2-tools 2024-02-19T21:55:28.730 INFO:teuthology.orchestra.run.smithi176.stdout:Name : grub2-tools 2024-02-19T21:55:28.730 INFO:teuthology.orchestra.run.smithi176.stdout:Epoch : 1 2024-02-19T21:55:28.730 INFO:teuthology.orchestra.run.smithi176.stdout:Version : 2.02 2024-02-19T21:55:28.730 INFO:teuthology.orchestra.run.smithi176.stdout:Release : 129.el8 2024-02-19T21:55:28.730 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: x86_64 2024-02-19T21:55:28.731 INFO:teuthology.orchestra.run.smithi176.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T21:55:28.731 INFO:teuthology.orchestra.run.smithi176.stdout:Group : System Environment/Base 2024-02-19T21:55:28.731 INFO:teuthology.orchestra.run.smithi176.stdout:Size : 9265586 2024-02-19T21:55:28.731 INFO:teuthology.orchestra.run.smithi176.stdout:License : GPLv3+ 2024-02-19T21:55:28.731 INFO:teuthology.orchestra.run.smithi176.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T21:55:28.731 INFO:teuthology.orchestra.run.smithi176.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T21:55:28.731 INFO:teuthology.orchestra.run.smithi176.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T21:55:28.731 INFO:teuthology.orchestra.run.smithi176.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T21:55:28.731 INFO:teuthology.orchestra.run.smithi176.stdout:Relocations : (not relocatable) 2024-02-19T21:55:28.732 INFO:teuthology.orchestra.run.smithi176.stdout:Packager : CentOS BuildSystem 2024-02-19T21:55:28.732 INFO:teuthology.orchestra.run.smithi176.stdout:Vendor : CentOS 2024-02-19T21:55:28.732 INFO:teuthology.orchestra.run.smithi176.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T21:55:28.732 INFO:teuthology.orchestra.run.smithi176.stdout:Summary : Support tools for GRUB. 2024-02-19T21:55:28.732 INFO:teuthology.orchestra.run.smithi176.stdout:Description : 2024-02-19T21:55:28.732 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T21:55:28.732 INFO:teuthology.orchestra.run.smithi176.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T21:55:28.732 INFO:teuthology.orchestra.run.smithi176.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T21:55:28.733 INFO:teuthology.orchestra.run.smithi176.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T21:55:28.733 INFO:teuthology.orchestra.run.smithi176.stdout:hardware devices. 2024-02-19T21:55:28.733 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T21:55:28.733 INFO:teuthology.orchestra.run.smithi176.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T21:55:28.734 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T21:55:28.734 INFO:teuthology.task.kernel:Updating grub on smithi176 to boot 4.18.0-539.el8.x86_64 2024-02-19T21:55:28.734 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T21:55:28.936 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:55:28.936 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:55:28.936 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:55:29.359 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:55:29.359 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T21:55:29.359 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T21:55:29.359 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T21:55:29.359 INFO:teuthology.orchestra.run.smithi184.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T21:55:29.360 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T21:55:29.360 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T21:55:29.515 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:55:29.735 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:55:29.735 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:55:29.735 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T21:55:29.735 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T21:55:29.735 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T21:55:29.735 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-19T21:55:29.736 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi184.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T21:55:29.736 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:55:29.946 INFO:teuthology.orchestra.run.smithi184.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:55:29.946 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:55:29.946 DEBUG:teuthology.orchestra.run.smithi184:> sudo rpm -qi grub2-tools 2024-02-19T21:55:29.979 INFO:teuthology.orchestra.run.smithi184.stdout:Name : grub2-tools 2024-02-19T21:55:29.979 INFO:teuthology.orchestra.run.smithi184.stdout:Epoch : 1 2024-02-19T21:55:29.979 INFO:teuthology.orchestra.run.smithi184.stdout:Version : 2.02 2024-02-19T21:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout:Release : 129.el8 2024-02-19T21:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: x86_64 2024-02-19T21:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T21:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout:Group : System Environment/Base 2024-02-19T21:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout:Size : 9265586 2024-02-19T21:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout:License : GPLv3+ 2024-02-19T21:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T21:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T21:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T21:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T21:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout:Relocations : (not relocatable) 2024-02-19T21:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout:Packager : CentOS BuildSystem 2024-02-19T21:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout:Vendor : CentOS 2024-02-19T21:55:29.980 INFO:teuthology.orchestra.run.smithi184.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T21:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout:Summary : Support tools for GRUB. 2024-02-19T21:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout:Description : 2024-02-19T21:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T21:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T21:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T21:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T21:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout:hardware devices. 2024-02-19T21:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T21:55:29.981 INFO:teuthology.orchestra.run.smithi184.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T21:55:29.982 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T21:55:29.982 INFO:teuthology.task.kernel:Updating grub on smithi184 to boot 4.18.0-539.el8.x86_64 2024-02-19T21:55:29.982 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T21:55:30.388 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2024-02-19T21:55:31.773 INFO:teuthology.orchestra.run.smithi184.stderr:Generating grub configuration file ... 2024-02-19T21:55:32.613 INFO:teuthology.orchestra.run.smithi176.stderr:done 2024-02-19T21:55:32.615 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-02-19T21:55:32.646 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.O29AT2MvA9 2024-02-19T21:55:32.647 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.O29AT2MvA9 2024-02-19T21:55:32.732 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.O29AT2MvA9 2024-02-19T21:55:32.828 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.O29AT2MvA9 is 6KB 2024-02-19T21:55:32.876 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.O29AT2MvA9 2024-02-19T21:55:32.893 DEBUG:teuthology.orchestra.run.smithi176:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T21:55:32.970 INFO:teuthology.orchestra.run.smithi176.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T21:55:32.970 INFO:teuthology.orchestra.run.smithi176.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T21:55:32.970 INFO:teuthology.orchestra.run.smithi176.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T21:55:32.970 INFO:teuthology.orchestra.run.smithi176.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T21:55:32.971 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T21:55:33.149 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2024-02-19T21:55:33.968 INFO:teuthology.orchestra.run.smithi184.stderr:done 2024-02-19T21:55:33.969 DEBUG:teuthology.orchestra.run.smithi184:> mktemp 2024-02-19T21:55:34.003 INFO:teuthology.orchestra.run.smithi184.stdout:/tmp/tmp.qdIOo7ncHA 2024-02-19T21:55:34.003 DEBUG:teuthology.orchestra.run.smithi184:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qdIOo7ncHA 2024-02-19T21:55:34.108 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 0666 /tmp/tmp.qdIOo7ncHA 2024-02-19T21:55:34.214 DEBUG:teuthology.orchestra.remote:smithi184:/tmp/tmp.qdIOo7ncHA is 6KB 2024-02-19T21:55:34.263 DEBUG:teuthology.orchestra.run.smithi184:> rm -fr /tmp/tmp.qdIOo7ncHA 2024-02-19T21:55:34.280 DEBUG:teuthology.orchestra.run.smithi184:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T21:55:34.371 INFO:teuthology.orchestra.run.smithi184.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T21:55:34.372 INFO:teuthology.orchestra.run.smithi184.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T21:55:34.372 INFO:teuthology.orchestra.run.smithi184.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T21:55:34.372 INFO:teuthology.orchestra.run.smithi184.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T21:55:34.375 DEBUG:teuthology.orchestra.run.smithi184:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T21:55:34.494 DEBUG:teuthology.orchestra.run.smithi184:> sudo shutdown -r now 2024-02-19T21:56:03.180 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-19T21:56:03.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-19T21:56:03.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:56:04.498 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-19T21:56:04.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-02-19T21:56:04.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:56:21.696 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-02-19T21:56:22.976 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-02-19T21:56:30.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-19T21:56:30.705 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:56:31.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-02-19T21:56:31.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:56:33.760 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-02-19T21:56:35.040 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.184 2024-02-19T21:56:45.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-19T21:56:45.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:56:47.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-02-19T21:56:47.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:57:18.586 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-02-19T21:57:18.903 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-19T21:57:18.903 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-539.el8.x86_64"... 2024-02-19T21:57:18.903 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-02-19T21:57:18.958 INFO:teuthology.orchestra.run.smithi176.stdout:4.18.0-539.el8.x86_64 2024-02-19T21:57:18.958 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T21:57:18.958 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T21:57:18.958 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T21:57:19.597 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-02-19T21:57:19.959 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-19T21:57:19.960 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T21:57:20.040 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2024-02-19T21:57:20.051 DEBUG:teuthology.parallel:result is None 2024-02-19T21:57:20.333 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-02-19T21:57:20.333 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-539.el8.x86_64"... 2024-02-19T21:57:20.334 DEBUG:teuthology.orchestra.run.smithi184:> uname -r 2024-02-19T21:57:20.350 INFO:teuthology.orchestra.run.smithi184.stdout:4.18.0-539.el8.x86_64 2024-02-19T21:57:20.350 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T21:57:20.350 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T21:57:20.350 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T21:57:21.351 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-19T21:57:21.352 DEBUG:teuthology.orchestra.run.smithi184:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T21:57:21.409 INFO:teuthology.orchestra.run.smithi184.stdout:ttyS1 2024-02-19T21:57:21.420 DEBUG:teuthology.parallel:result is None 2024-02-19T21:57:21.421 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-19T21:57:21.429 INFO:teuthology.task.internal:Creating test directory... 2024-02-19T21:57:21.430 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T21:57:21.433 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T21:57:21.452 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-19T21:57:21.478 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-19T21:57:21.528 INFO:teuthology.task.internal:Creating archive directory... 2024-02-19T21:57:21.528 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T21:57:21.531 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T21:57:21.576 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-19T21:57:21.583 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-19T21:57:21.583 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T21:57:21.610 DEBUG:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T21:57:21.646 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:57:21.656 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:57:21.656 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:57:21.668 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:57:21.669 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-19T21:57:21.676 INFO:teuthology.task.internal:Configuring sudo... 2024-02-19T21:57:21.677 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T21:57:21.679 DEBUG:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T21:57:21.725 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-19T21:57:21.736 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-19T21:57:21.736 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T21:57:21.749 DEBUG:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T21:57:21.780 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:57:21.855 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:57:21.916 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:57:21.962 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:57:22.004 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-19T21:57:22.005 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T21:57:22.071 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:57:22.120 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:57:22.166 DEBUG:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:57:22.251 DEBUG:teuthology.orchestra.run.smithi184:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:57:22.289 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-02-19T21:57:22.289 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T21:57:22.353 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-02-19T21:57:22.355 DEBUG:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2024-02-19T21:57:22.409 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T21:57:22.455 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T21:57:22.850 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-19T21:57:22.859 INFO:teuthology.task.internal:Starting timer... 2024-02-19T21:57:22.859 INFO:teuthology.run_tasks:Running task pcp... 2024-02-19T21:57:22.914 INFO:teuthology.run_tasks:Running task selinux... 2024-02-19T21:57:22.962 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-19T21:57:22.963 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2024-02-19T21:57:23.024 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2024-02-19T21:57:23.026 DEBUG:teuthology.orchestra.run.smithi184:> sudo service auditd rotate 2024-02-19T21:57:23.085 INFO:teuthology.orchestra.run.smithi184.stdout:Rotating logs: 2024-02-19T21:57:23.088 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-19T21:57:23.088 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2024-02-19T21:57:23.135 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2024-02-19T21:57:23.136 DEBUG:teuthology.orchestra.run.smithi184:> /usr/sbin/getenforce 2024-02-19T21:57:23.160 INFO:teuthology.orchestra.run.smithi184.stdout:Permissive 2024-02-19T21:57:23.160 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi176.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi184.front.sepia.ceph.com': 'permissive'} 2024-02-19T21:57:23.161 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T21:57:23.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:57:23.187 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T21:57:23.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:57:23.228 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-19T21:57:23.228 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2024-02-19T21:57:23.262 DEBUG:teuthology.orchestra.run.smithi184:> sudo /usr/sbin/setenforce permissive 2024-02-19T21:57:23.295 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-19T21:57:23.304 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-19T21:57:23.322 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-19T21:57:23.323 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi176.front.sepia.ceph.com,smithi184.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-19T22:03:21.912 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi176.front.sepia.ceph.com'), Remote(name='ubuntu@smithi184.front.sepia.ceph.com')] 2024-02-19T22:03:21.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-19T22:03:21.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:03:21.993 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-02-19T22:03:22.067 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-19T22:03:22.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-02-19T22:03:22.068 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:03:22.143 DEBUG:teuthology.orchestra.run.smithi184:> true 2024-02-19T22:03:22.210 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi184.front.sepia.ceph.com' 2024-02-19T22:03:22.210 INFO:teuthology.run_tasks:Running task clock... 2024-02-19T22:03:22.221 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-19T22:03:22.221 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T22:03:22.221 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:03:22.224 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T22:03:22.224 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:03:22.255 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T22:03:22.269 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T22:03:22.282 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T22:03:22.295 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T22:03:22.333 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2024-02-19T22:03:22.344 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2024-02-19T22:03:22.357 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T22:03:22.363 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: ntpd: command not found 2024-02-19T22:03:22.370 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T22:03:22.374 INFO:teuthology.orchestra.run.smithi184.stdout:506 Cannot talk to daemon 2024-02-19T22:03:22.387 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T22:03:22.400 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T22:03:22.439 INFO:teuthology.orchestra.run.smithi176.stderr:bash: ntpq: command not found 2024-02-19T22:03:22.443 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:03:22.443 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-02-19T22:03:22.443 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:03:22.443 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:03:22.443 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:03:22.443 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:03:22.462 INFO:teuthology.orchestra.run.smithi184.stderr:bash: ntpq: command not found 2024-02-19T22:03:22.466 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:03:22.466 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-02-19T22:03:22.466 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:03:22.466 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:03:22.466 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:03:22.466 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:03:22.467 INFO:teuthology.run_tasks:Running task pexec... 2024-02-19T22:03:22.479 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-19T22:03:22.480 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi176.front.sepia.ceph.com 2024-02-19T22:03:22.480 DEBUG:teuthology.orchestra.run.smithi176:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:03:22.480 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi184.front.sepia.ceph.com 2024-02-19T22:03:22.481 DEBUG:teuthology.orchestra.run.smithi184:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:03:23.269 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:29 ago on Mon 19 Feb 2024 10:01:54 PM UTC. 2024-02-19T22:03:23.293 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Feb 2024 10:01:48 PM UTC. 2024-02-19T22:03:25.322 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:03:25.323 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:03:25.323 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-02-19T22:03:25.323 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:03:25.323 INFO:teuthology.orchestra.run.smithi176.stdout:Resetting modules: 2024-02-19T22:03:25.323 INFO:teuthology.orchestra.run.smithi176.stdout: container-tools 2024-02-19T22:03:25.323 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:03:25.323 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:03:25.324 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:03:25.324 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:03:25.352 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:03:25.353 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:03:25.353 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-02-19T22:03:25.354 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:03:25.354 INFO:teuthology.orchestra.run.smithi184.stdout:Resetting modules: 2024-02-19T22:03:25.354 INFO:teuthology.orchestra.run.smithi184.stdout: container-tools 2024-02-19T22:03:25.354 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:03:25.354 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:03:25.354 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:03:25.354 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:03:25.572 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:03:25.620 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:03:26.425 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:32 ago on Mon 19 Feb 2024 10:01:54 PM UTC. 2024-02-19T22:03:26.490 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:38 ago on Mon 19 Feb 2024 10:01:48 PM UTC. 2024-02-19T22:03:28.462 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:03:28.473 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:03:28.497 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:03:28.497 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-02-19T22:03:28.497 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:03:28.497 INFO:teuthology.orchestra.run.smithi176.stdout:Installing group/module packages: 2024-02-19T22:03:28.497 INFO:teuthology.orchestra.run.smithi176.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-19T22:03:28.497 INFO:teuthology.orchestra.run.smithi176.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-19T22:03:28.497 INFO:teuthology.orchestra.run.smithi176.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-19T22:03:28.497 INFO:teuthology.orchestra.run.smithi176.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-19T22:03:28.497 INFO:teuthology.orchestra.run.smithi176.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-19T22:03:28.498 INFO:teuthology.orchestra.run.smithi176.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-19T22:03:28.498 INFO:teuthology.orchestra.run.smithi176.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-19T22:03:28.498 INFO:teuthology.orchestra.run.smithi176.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-19T22:03:28.498 INFO:teuthology.orchestra.run.smithi176.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-19T22:03:28.498 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-02-19T22:03:28.498 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-19T22:03:28.498 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-19T22:03:28.498 INFO:teuthology.orchestra.run.smithi176.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-19T22:03:28.498 INFO:teuthology.orchestra.run.smithi176.stdout:Installing weak dependencies: 2024-02-19T22:03:28.498 INFO:teuthology.orchestra.run.smithi176.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-19T22:03:28.498 INFO:teuthology.orchestra.run.smithi176.stdout:Installing module profiles: 2024-02-19T22:03:28.498 INFO:teuthology.orchestra.run.smithi176.stdout: container-tools/common 2024-02-19T22:03:28.498 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:03:28.498 INFO:teuthology.orchestra.run.smithi176.stdout:Enabling module streams: 2024-02-19T22:03:28.499 INFO:teuthology.orchestra.run.smithi176.stdout: container-tools rhel8 2024-02-19T22:03:28.499 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:03:28.499 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:03:28.499 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:03:28.499 INFO:teuthology.orchestra.run.smithi176.stdout:Install 13 Packages 2024-02-19T22:03:28.499 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:03:28.499 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 28 M 2024-02-19T22:03:28.499 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 92 M 2024-02-19T22:03:28.500 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:03:28.508 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:03:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-02-19T22:03:28.508 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:03:28.508 INFO:teuthology.orchestra.run.smithi184.stdout:Installing group/module packages: 2024-02-19T22:03:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-19T22:03:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-19T22:03:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-19T22:03:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-19T22:03:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-19T22:03:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-19T22:03:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-19T22:03:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-19T22:03:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-19T22:03:28.509 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-02-19T22:03:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-19T22:03:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-19T22:03:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-19T22:03:28.509 INFO:teuthology.orchestra.run.smithi184.stdout:Installing weak dependencies: 2024-02-19T22:03:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-19T22:03:28.509 INFO:teuthology.orchestra.run.smithi184.stdout:Installing module profiles: 2024-02-19T22:03:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: container-tools/common 2024-02-19T22:03:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:03:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:Enabling module streams: 2024-02-19T22:03:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: container-tools rhel8 2024-02-19T22:03:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:03:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:03:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:03:28.510 INFO:teuthology.orchestra.run.smithi184.stdout:Install 13 Packages 2024-02-19T22:03:28.510 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:03:28.511 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 28 M 2024-02-19T22:03:28.511 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 92 M 2024-02-19T22:03:28.511 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:03:29.002 INFO:teuthology.orchestra.run.smithi184.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 2.3 MB/s | 688 kB 00:00 2024-02-19T22:03:29.036 INFO:teuthology.orchestra.run.smithi176.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 2.0 MB/s | 688 kB 00:00 2024-02-19T22:03:29.085 INFO:teuthology.orchestra.run.smithi184.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 455 kB/s | 38 kB 00:00 2024-02-19T22:03:29.095 INFO:teuthology.orchestra.run.smithi176.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 651 kB/s | 38 kB 00:00 2024-02-19T22:03:29.119 INFO:teuthology.orchestra.run.smithi184.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.5 MB/s | 1.0 MB 00:00 2024-02-19T22:03:29.212 INFO:teuthology.orchestra.run.smithi176.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.9 MB/s | 1.0 MB 00:00 2024-02-19T22:03:29.219 INFO:teuthology.orchestra.run.smithi184.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 1.9 MB/s | 257 kB 00:00 2024-02-19T22:03:29.254 INFO:teuthology.orchestra.run.smithi176.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 1.6 MB/s | 257 kB 00:00 2024-02-19T22:03:29.336 INFO:teuthology.orchestra.run.smithi184.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.5 MB/s | 180 kB 00:00 2024-02-19T22:03:29.371 INFO:teuthology.orchestra.run.smithi176.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.5 MB/s | 180 kB 00:00 2024-02-19T22:03:29.395 INFO:teuthology.orchestra.run.smithi184.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 425 kB/s | 25 kB 00:00 2024-02-19T22:03:29.413 INFO:teuthology.orchestra.run.smithi176.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 595 kB/s | 25 kB 00:00 2024-02-19T22:03:29.454 INFO:teuthology.orchestra.run.smithi184.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2024-02-19T22:03:29.514 INFO:teuthology.orchestra.run.smithi176.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 934 kB/s | 94 kB 00:00 2024-02-19T22:03:31.421 INFO:teuthology.orchestra.run.smithi184.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 1.9 MB/s | 4.3 MB 00:02 2024-02-19T22:03:31.872 INFO:teuthology.orchestra.run.smithi176.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 1.6 MB/s | 4.3 MB 00:02 2024-02-19T22:03:32.996 INFO:teuthology.orchestra.run.smithi184.stdout:(9/13): toolbox-0.0.99.5-1.module_el8+847+7863d 1.6 MB/s | 2.5 MB 00:01 2024-02-19T22:03:33.054 INFO:teuthology.orchestra.run.smithi184.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 908 kB/s | 53 kB 00:00 2024-02-19T22:03:33.130 INFO:teuthology.orchestra.run.smithi184.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 545 kB/s | 41 kB 00:00 2024-02-19T22:03:33.755 INFO:teuthology.orchestra.run.smithi176.stdout:(9/13): toolbox-0.0.99.5-1.module_el8+847+7863d 1.3 MB/s | 2.5 MB 00:01 2024-02-19T22:03:33.805 INFO:teuthology.orchestra.run.smithi176.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 1.0 MB/s | 53 kB 00:00 2024-02-19T22:03:33.847 INFO:teuthology.orchestra.run.smithi176.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 984 kB/s | 41 kB 00:00 2024-02-19T22:03:34.030 INFO:teuthology.orchestra.run.smithi184.stdout:(12/13): buildah-1.33.5-1.module_el8+885+7da147 1.8 MB/s | 9.7 MB 00:05 2024-02-19T22:03:34.839 INFO:teuthology.orchestra.run.smithi176.stdout:(12/13): buildah-1.33.5-1.module_el8+885+7da147 1.6 MB/s | 9.7 MB 00:06 2024-02-19T22:03:35.372 INFO:teuthology.orchestra.run.smithi176.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 1.5 MB/s | 8.8 MB 00:05 2024-02-19T22:03:35.373 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:03:35.373 INFO:teuthology.orchestra.run.smithi176.stdout:Total 4.0 MB/s | 28 MB 00:06 2024-02-19T22:03:35.401 INFO:teuthology.orchestra.run.smithi184.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 1.5 MB/s | 8.8 MB 00:05 2024-02-19T22:03:35.402 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:03:35.402 INFO:teuthology.orchestra.run.smithi184.stdout:Total 4.0 MB/s | 28 MB 00:06 2024-02-19T22:03:35.645 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:03:35.666 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:03:35.666 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:03:35.675 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:03:35.695 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:03:35.695 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:03:35.928 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:03:35.932 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:03:35.969 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:03:35.972 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:03:36.615 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:03:36.643 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:03:37.575 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-19T22:03:37.577 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-19T22:03:37.780 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:03:37.794 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:03:37.964 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-19T22:03:37.983 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-19T22:03:38.148 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-19T22:03:38.170 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-19T22:03:38.400 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-19T22:03:38.402 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-19T22:03:38.986 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-19T22:03:38.997 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-19T22:03:39.935 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-19T22:03:40.001 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-19T22:03:40.179 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-19T22:03:40.399 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-19T22:03:40.488 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-19T22:03:40.684 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-19T22:03:40.840 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-19T22:03:41.095 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:03:41.116 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-19T22:03:41.329 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:03:41.334 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:03:41.521 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:03:41.533 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:03:41.741 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:03:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:03:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-19T22:03:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:03:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-19T22:03:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-19T22:03:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-19T22:03:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-19T22:03:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-19T22:03:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-19T22:03:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-19T22:03:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-19T22:03:41.928 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:03:41.929 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:03:42.169 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:03:42.169 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-19T22:03:42.169 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:03:42.169 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-19T22:03:42.170 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-19T22:03:42.170 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-19T22:03:42.170 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-19T22:03:42.170 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-19T22:03:42.170 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-19T22:03:42.170 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-19T22:03:42.170 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-19T22:03:42.170 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:03:42.170 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:03:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-19T22:03:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:03:42.429 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:03:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-19T22:03:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:03:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:03:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-19T22:03:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-19T22:03:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:03:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:03:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-19T22:03:42.429 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-19T22:03:42.430 INFO:teuthology.orchestra.run.smithi184.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-19T22:03:42.430 INFO:teuthology.orchestra.run.smithi184.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-19T22:03:42.430 INFO:teuthology.orchestra.run.smithi184.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-19T22:03:42.430 INFO:teuthology.orchestra.run.smithi184.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-19T22:03:42.430 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:03:42.430 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:03:42.701 DEBUG:teuthology.parallel:result is None 2024-02-19T22:03:42.707 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-19T22:03:42.707 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:03:42.708 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:03:42.708 INFO:teuthology.orchestra.run.smithi176.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-19T22:03:42.708 INFO:teuthology.orchestra.run.smithi176.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:03:42.708 INFO:teuthology.orchestra.run.smithi176.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:03:42.708 INFO:teuthology.orchestra.run.smithi176.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-19T22:03:42.708 INFO:teuthology.orchestra.run.smithi176.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-19T22:03:42.708 INFO:teuthology.orchestra.run.smithi176.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:03:42.708 INFO:teuthology.orchestra.run.smithi176.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:03:42.708 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-19T22:03:42.708 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-19T22:03:42.708 INFO:teuthology.orchestra.run.smithi176.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-19T22:03:42.708 INFO:teuthology.orchestra.run.smithi176.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-19T22:03:42.708 INFO:teuthology.orchestra.run.smithi176.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-19T22:03:42.708 INFO:teuthology.orchestra.run.smithi176.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-19T22:03:42.709 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:03:42.709 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:03:42.978 DEBUG:teuthology.parallel:result is None 2024-02-19T22:03:42.979 INFO:teuthology.run_tasks:Running task pexec... 2024-02-19T22:03:42.987 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-19T22:03:42.987 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi176.front.sepia.ceph.com 2024-02-19T22:03:42.987 DEBUG:teuthology.orchestra.run.smithi176:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:03:42.988 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi184.front.sepia.ceph.com 2024-02-19T22:03:42.988 DEBUG:teuthology.orchestra.run.smithi184:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:03:43.184 DEBUG:teuthology.parallel:result is None 2024-02-19T22:03:43.187 DEBUG:teuthology.parallel:result is None 2024-02-19T22:03:43.187 INFO:teuthology.run_tasks:Running task install... 2024-02-19T22:03:43.196 DEBUG:teuthology.task.install:project ceph 2024-02-19T22:03:43.196 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'}} 2024-02-19T22:03:43.196 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'} 2024-02-19T22:03:43.196 INFO:teuthology.task.install:Using flavor: default 2024-02-19T22:03:43.204 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-19T22:03:43.204 INFO:teuthology.task.install:extra packages: [] 2024-02-19T22:03:43.204 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-19T22:03:43.204 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:03:43.204 INFO:teuthology.packaging:ref: None 2024-02-19T22:03:43.204 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:03:43.204 INFO:teuthology.packaging:branch: None 2024-02-19T22:03:43.205 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:03:43.481 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:03:43.482 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:03:43.485 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-19T22:03:43.485 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:03:43.485 INFO:teuthology.packaging:ref: None 2024-02-19T22:03:43.485 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:03:43.485 INFO:teuthology.packaging:branch: None 2024-02-19T22:03:43.485 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:03:43.485 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:03:43.630 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-19T22:03:43.630 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-19T22:03:43.642 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-19T22:03:43.642 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-19T22:03:43.670 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T22:03:43.670 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-19T22:03:43.670 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T22:03:43.672 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T22:03:43.672 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-02-19T22:03:43.672 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T22:03:43.699 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T22:03:43.699 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:03:43.699 INFO:teuthology.packaging:ref: None 2024-02-19T22:03:43.699 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:03:43.699 INFO:teuthology.packaging:branch: None 2024-02-19T22:03:43.699 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:03:43.700 DEBUG:teuthology.orchestra.run.smithi176:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T22:03:43.701 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T22:03:43.701 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:03:43.701 INFO:teuthology.packaging:ref: None 2024-02-19T22:03:43.701 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:03:43.701 INFO:teuthology.packaging:branch: None 2024-02-19T22:03:43.701 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:03:43.701 DEBUG:teuthology.orchestra.run.smithi184:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T22:03:43.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T22:03:43.770 DEBUG:teuthology.orchestra.run.smithi184:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T22:03:43.844 DEBUG:teuthology.orchestra.run.smithi176:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T22:03:43.846 DEBUG:teuthology.orchestra.run.smithi184:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T22:03:43.910 INFO:teuthology.orchestra.run.smithi176.stdout:check_obsoletes = 1 2024-02-19T22:03:43.911 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum clean all 2024-02-19T22:03:43.912 INFO:teuthology.orchestra.run.smithi184.stdout:check_obsoletes = 1 2024-02-19T22:03:43.914 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum clean all 2024-02-19T22:03:44.178 INFO:teuthology.orchestra.run.smithi176.stdout:76 files removed 2024-02-19T22:03:44.181 INFO:teuthology.orchestra.run.smithi184.stdout:76 files removed 2024-02-19T22:03:44.206 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-radosgw 2024-02-19T22:03:44.212 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-radosgw 2024-02-19T22:03:44.689 INFO:teuthology.orchestra.run.smithi176.stdout:ceph packages for x86_64 962 kB/s | 74 kB 00:00 2024-02-19T22:03:44.706 INFO:teuthology.orchestra.run.smithi184.stdout:ceph packages for x86_64 838 kB/s | 74 kB 00:00 2024-02-19T22:03:44.815 INFO:teuthology.orchestra.run.smithi184.stdout:ceph noarch packages 213 kB/s | 15 kB 00:00 2024-02-19T22:03:44.863 INFO:teuthology.orchestra.run.smithi176.stdout:ceph noarch packages 111 kB/s | 15 kB 00:00 2024-02-19T22:03:44.907 INFO:teuthology.orchestra.run.smithi184.stdout:ceph source packages 27 kB/s | 1.7 kB 00:00 2024-02-19T22:03:44.973 INFO:teuthology.orchestra.run.smithi176.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2024-02-19T22:03:58.131 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - AppStream 2.1 MB/s | 27 MB 00:13 2024-02-19T22:04:00.198 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 8 - AppStream 1.8 MB/s | 27 MB 00:15 2024-02-19T22:04:04.248 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - BaseOS 8.0 MB/s | 10 MB 00:01 2024-02-19T22:04:05.899 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 8 - BaseOS 12 MB/s | 10 MB 00:00 2024-02-19T22:04:05.983 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - Extras 119 kB/s | 18 kB 00:00 2024-02-19T22:04:06.224 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - Extras common packages 52 kB/s | 6.9 kB 00:00 2024-02-19T22:04:07.600 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 8 - PowerTools 3.8 MB/s | 4.8 MB 00:01 2024-02-19T22:04:07.782 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 8 - Extras 64 kB/s | 18 kB 00:00 2024-02-19T22:04:08.668 INFO:teuthology.orchestra.run.smithi176.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-02-19T22:04:09.224 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 8 - Extras common packages 5.2 kB/s | 6.9 kB 00:01 2024-02-19T22:04:10.204 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-02-19T22:04:10.659 INFO:teuthology.orchestra.run.smithi184.stdout:CentOS Stream 8 - PowerTools 3.6 MB/s | 4.8 MB 00:01 2024-02-19T22:04:12.204 INFO:teuthology.orchestra.run.smithi184.stdout:Copr repo for python3-asyncssh owned by ceph 5.6 kB/s | 3.5 kB 00:00 2024-02-19T22:04:14.003 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 441 kB/s | 24 kB 00:00 2024-02-19T22:04:14.912 INFO:teuthology.orchestra.run.smithi184.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 16 MB 00:02 2024-02-19T22:04:17.499 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:04:17.502 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:04:17.502 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-02-19T22:04:17.502 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:04:17.502 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:04:17.502 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-19T22:04:17.502 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrading: 2024-02-19T22:04:17.502 INFO:teuthology.orchestra.run.smithi176.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:04:17.502 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-19T22:04:17.503 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-02-19T22:04:17.503 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-19T22:04:17.503 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-19T22:04:17.503 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-19T22:04:17.503 INFO:teuthology.orchestra.run.smithi176.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-19T22:04:17.503 INFO:teuthology.orchestra.run.smithi176.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-19T22:04:17.503 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-19T22:04:17.503 INFO:teuthology.orchestra.run.smithi176.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-19T22:04:17.503 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-19T22:04:17.503 INFO:teuthology.orchestra.run.smithi176.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-19T22:04:17.503 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-19T22:04:17.503 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-19T22:04:17.504 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-19T22:04:17.504 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-19T22:04:17.504 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-19T22:04:17.504 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-19T22:04:17.504 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-19T22:04:17.504 INFO:teuthology.orchestra.run.smithi176.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-19T22:04:17.504 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-19T22:04:17.504 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-19T22:04:17.504 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-19T22:04:17.504 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-19T22:04:17.504 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-19T22:04:17.505 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-19T22:04:17.505 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-19T22:04:17.505 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:04:17.505 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:04:17.505 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:04:17.505 INFO:teuthology.orchestra.run.smithi176.stdout:Install 25 Packages 2024-02-19T22:04:17.505 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrade 2 Packages 2024-02-19T22:04:17.505 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:04:17.506 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 59 M 2024-02-19T22:04:17.506 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:04:17.887 INFO:teuthology.orchestra.run.smithi176.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 51 MB/s | 6.0 MB 00:00 2024-02-19T22:04:17.912 INFO:teuthology.orchestra.run.smithi176.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-02-19T22:04:17.946 INFO:teuthology.orchestra.run.smithi176.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2024-02-19T22:04:17.980 INFO:teuthology.orchestra.run.smithi176.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-19T22:04:18.072 INFO:teuthology.orchestra.run.smithi176.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 44 MB/s | 4.1 MB 00:00 2024-02-19T22:04:18.089 INFO:teuthology.orchestra.run.smithi176.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.7 MB/s | 44 kB 00:00 2024-02-19T22:04:18.139 INFO:teuthology.orchestra.run.smithi176.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 1.5 MB/s | 80 kB 00:00 2024-02-19T22:04:18.173 INFO:teuthology.orchestra.run.smithi176.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.3 MB/s | 213 kB 00:00 2024-02-19T22:04:18.198 INFO:teuthology.orchestra.run.smithi176.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 15 MB/s | 387 kB 00:00 2024-02-19T22:04:18.232 INFO:teuthology.orchestra.run.smithi176.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-02-19T22:04:18.273 INFO:teuthology.orchestra.run.smithi176.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 114 kB 00:00 2024-02-19T22:04:18.440 INFO:teuthology.orchestra.run.smithi176.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 19 MB/s | 13 MB 00:00 2024-02-19T22:04:18.491 INFO:teuthology.orchestra.run.smithi176.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 669 kB/s | 145 kB 00:00 2024-02-19T22:04:18.712 INFO:teuthology.orchestra.run.smithi184.stdout:lab-extras 299 kB/s | 24 kB 00:00 2024-02-19T22:04:18.716 INFO:teuthology.orchestra.run.smithi176.stdout:(14/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2024-02-19T22:04:18.791 INFO:teuthology.orchestra.run.smithi176.stdout:(15/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 865 kB/s | 259 kB 00:00 2024-02-19T22:04:18.825 INFO:teuthology.orchestra.run.smithi176.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 405 kB/s | 44 kB 00:00 2024-02-19T22:04:18.867 INFO:teuthology.orchestra.run.smithi176.stdout:(17/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.6 MB/s | 677 kB 00:00 2024-02-19T22:04:19.026 INFO:teuthology.orchestra.run.smithi176.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 180 kB/s | 42 kB 00:00 2024-02-19T22:04:19.051 INFO:teuthology.orchestra.run.smithi176.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 209 kB/s | 47 kB 00:00 2024-02-19T22:04:19.079 INFO:teuthology.orchestra.run.smithi176.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 383 kB/s | 80 kB 00:00 2024-02-19T22:04:19.210 INFO:teuthology.orchestra.run.smithi176.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 633 kB/s | 116 kB 00:00 2024-02-19T22:04:19.252 INFO:teuthology.orchestra.run.smithi176.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 1.1 MB/s | 181 kB 00:00 2024-02-19T22:04:19.277 INFO:teuthology.orchestra.run.smithi176.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 880 kB/s | 59 kB 00:00 2024-02-19T22:04:19.302 INFO:teuthology.orchestra.run.smithi176.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2024-02-19T22:04:19.327 INFO:teuthology.orchestra.run.smithi176.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 995 kB/s | 75 kB 00:00 2024-02-19T22:04:19.428 INFO:teuthology.orchestra.run.smithi176.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 25 MB/s | 3.7 MB 00:00 2024-02-19T22:04:19.495 INFO:teuthology.orchestra.run.smithi176.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 21 MB/s | 4.0 MB 00:00 2024-02-19T22:04:19.500 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:19.500 INFO:teuthology.orchestra.run.smithi176.stdout:Total 30 MB/s | 59 MB 00:01 2024-02-19T22:04:19.569 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:04:19.617 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:04:19.618 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:04:19.963 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:04:19.967 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:04:21.100 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:04:21.366 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-19T22:04:21.572 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-19T22:04:21.752 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:04:22.148 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:04:22.283 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:04:22.286 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:04:22.286 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-02-19T22:04:22.287 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:04:22.287 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:04:22.287 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-19T22:04:22.287 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrading: 2024-02-19T22:04:22.287 INFO:teuthology.orchestra.run.smithi184.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:04:22.287 INFO:teuthology.orchestra.run.smithi184.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-19T22:04:22.287 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-02-19T22:04:22.287 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-19T22:04:22.287 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-19T22:04:22.287 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-19T22:04:22.287 INFO:teuthology.orchestra.run.smithi184.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-19T22:04:22.287 INFO:teuthology.orchestra.run.smithi184.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-19T22:04:22.287 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-19T22:04:22.287 INFO:teuthology.orchestra.run.smithi184.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-19T22:04:22.288 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-19T22:04:22.288 INFO:teuthology.orchestra.run.smithi184.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-19T22:04:22.288 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-19T22:04:22.288 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-19T22:04:22.288 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-19T22:04:22.288 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-19T22:04:22.288 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-19T22:04:22.288 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-19T22:04:22.288 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-19T22:04:22.288 INFO:teuthology.orchestra.run.smithi184.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-19T22:04:22.288 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-19T22:04:22.288 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-19T22:04:22.288 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-19T22:04:22.289 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-19T22:04:22.289 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-19T22:04:22.289 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-19T22:04:22.289 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-19T22:04:22.289 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:04:22.289 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:04:22.289 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:04:22.289 INFO:teuthology.orchestra.run.smithi184.stdout:Install 25 Packages 2024-02-19T22:04:22.289 INFO:teuthology.orchestra.run.smithi184.stdout:Upgrade 2 Packages 2024-02-19T22:04:22.289 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:04:22.290 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 59 M 2024-02-19T22:04:22.290 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:04:22.335 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:04:22.412 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:04:22.691 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:04:22.860 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:04:22.907 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:04:22.972 INFO:teuthology.orchestra.run.smithi184.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 37 MB/s | 6.0 MB 00:00 2024-02-19T22:04:22.997 INFO:teuthology.orchestra.run.smithi184.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 996 kB/s | 25 kB 00:00 2024-02-19T22:04:23.039 INFO:teuthology.orchestra.run.smithi184.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-02-19T22:04:23.073 INFO:teuthology.orchestra.run.smithi184.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-19T22:04:23.119 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-19T22:04:23.165 INFO:teuthology.orchestra.run.smithi184.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 45 MB/s | 4.1 MB 00:00 2024-02-19T22:04:23.181 INFO:teuthology.orchestra.run.smithi184.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.6 MB/s | 44 kB 00:00 2024-02-19T22:04:23.198 INFO:teuthology.orchestra.run.smithi184.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 4.7 MB/s | 80 kB 00:00 2024-02-19T22:04:23.232 INFO:teuthology.orchestra.run.smithi184.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 6.2 MB/s | 213 kB 00:00 2024-02-19T22:04:23.283 INFO:teuthology.orchestra.run.smithi184.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 7.6 MB/s | 387 kB 00:00 2024-02-19T22:04:23.310 INFO:teuthology.orchestra.run.smithi184.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 13 MB/s | 368 kB 00:00 2024-02-19T22:04:23.335 INFO:teuthology.orchestra.run.smithi184.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-19T22:04:23.477 INFO:teuthology.orchestra.run.smithi184.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 19 MB/s | 13 MB 00:00 2024-02-19T22:04:23.677 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-19T22:04:23.744 INFO:teuthology.orchestra.run.smithi184.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 26 MB/s | 24 MB 00:00 2024-02-19T22:04:23.769 INFO:teuthology.orchestra.run.smithi184.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 335 kB/s | 145 kB 00:00 2024-02-19T22:04:23.812 INFO:teuthology.orchestra.run.smithi184.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 1.0 MB/s | 44 kB 00:00 2024-02-19T22:04:23.826 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:04:23.928 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:04:23.995 INFO:teuthology.orchestra.run.smithi184.stdout:(16/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2024-02-19T22:04:24.020 INFO:teuthology.orchestra.run.smithi184.stdout:(17/27): daxctl-libs-71.1-7.el8.x86_64.rpm 202 kB/s | 42 kB 00:00 2024-02-19T22:04:24.046 INFO:teuthology.orchestra.run.smithi184.stdout:(18/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.2 MB/s | 677 kB 00:00 2024-02-19T22:04:24.071 INFO:teuthology.orchestra.run.smithi184.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 623 kB/s | 47 kB 00:00 2024-02-19T22:04:24.123 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-19T22:04:24.238 INFO:teuthology.orchestra.run.smithi184.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 369 kB/s | 80 kB 00:00 2024-02-19T22:04:24.293 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:04:24.362 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:04:24.535 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:04:24.693 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:04:24.868 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:04:24.930 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:04:25.068 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:04:25.421 INFO:teuthology.orchestra.run.smithi184.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 85 kB/s | 116 kB 00:01 2024-02-19T22:04:25.482 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:04:25.651 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:04:25.735 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:04:25.938 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-19T22:04:25.946 INFO:teuthology.orchestra.run.smithi184.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 163 kB/s | 306 kB 00:01 2024-02-19T22:04:25.972 INFO:teuthology.orchestra.run.smithi184.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 104 kB/s | 181 kB 00:01 2024-02-19T22:04:25.997 INFO:teuthology.orchestra.run.smithi184.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 102 kB/s | 59 kB 00:00 2024-02-19T22:04:26.081 INFO:teuthology.orchestra.run.smithi184.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2024-02-19T22:04:26.097 INFO:teuthology.orchestra.run.smithi184.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 498 kB/s | 75 kB 00:00 2024-02-19T22:04:26.136 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-19T22:04:26.173 INFO:teuthology.orchestra.run.smithi184.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 23 MB/s | 4.0 MB 00:00 2024-02-19T22:04:26.175 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:26.175 INFO:teuthology.orchestra.run.smithi184.stdout:Total 15 MB/s | 59 MB 00:03 2024-02-19T22:04:26.243 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:04:26.284 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:04:26.291 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:04:26.291 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:04:26.360 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:04:26.568 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-19T22:04:26.649 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:04:26.652 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:04:26.788 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-19T22:04:26.941 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-19T22:04:27.143 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-19T22:04:27.789 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:04:28.017 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-19T22:04:28.229 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-19T22:04:28.389 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:04:28.840 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:04:29.056 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:04:29.139 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:04:29.372 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:04:29.523 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:04:29.588 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:04:29.617 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:04:29.815 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-19T22:04:29.867 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:04:30.357 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-19T22:04:30.489 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:04:30.570 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:04:30.579 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:04:30.777 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-19T22:04:30.860 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:04:30.939 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:04:30.953 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:04:30.990 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:04:31.091 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:04:31.161 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:04:31.357 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:04:31.530 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:04:31.593 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:04:31.722 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:04:32.155 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:04:32.338 INFO:teuthology.orchestra.run.smithi184.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:04:32.434 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:04:32.642 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-19T22:04:32.841 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-19T22:04:33.068 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:04:33.143 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:04:33.316 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-19T22:04:33.540 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-19T22:04:33.739 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-19T22:04:33.947 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-19T22:04:36.507 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:04:36.747 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:04:37.446 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:04:37.732 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:04:37.836 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:04:37.987 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:04:51.113 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:04:51.114 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /sys 2024-02-19T22:04:51.114 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /proc 2024-02-19T22:04:51.114 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev 2024-02-19T22:04:51.114 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /run 2024-02-19T22:04:51.114 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2024-02-19T22:04:51.114 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /var/tmp 2024-02-19T22:04:51.114 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /home 2024-02-19T22:04:51.114 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /tmp 2024-02-19T22:04:51.114 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /dev 2024-02-19T22:04:51.114 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:04:52.464 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:04:52.528 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:04:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:04:52.732 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:04:52.732 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:04:53.426 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:04:53.426 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-19T22:04:53.426 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-19T22:04:53.426 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-19T22:04:53.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:04:53.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:04:53.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:04:53.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-19T22:04:53.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-19T22:04:53.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:04:53.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-19T22:04:53.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-19T22:04:53.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:04:53.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:04:53.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-19T22:04:53.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:04:53.427 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-19T22:04:53.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-19T22:04:53.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-19T22:04:53.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-19T22:04:53.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-19T22:04:53.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-19T22:04:53.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-19T22:04:53.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-19T22:04:53.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-19T22:04:53.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-19T22:04:53.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:04:53.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-19T22:04:53.428 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-19T22:04:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:04:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:04:54.360 INFO:teuthology.orchestra.run.smithi176.stdout:Upgraded: 2024-02-19T22:04:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:54.360 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:04:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:54.360 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: leveldb-1.22-1.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-19T22:04:54.361 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-19T22:04:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:54.362 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:04:54.362 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:04:54.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-test 2024-02-19T22:04:55.467 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:42 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:04:57.026 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:04:57.027 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:04:57.027 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-02-19T22:04:57.027 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:04:57.027 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:04:57.027 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-19T22:04:57.027 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-02-19T22:04:57.027 INFO:teuthology.orchestra.run.smithi176.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-19T22:04:57.028 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-19T22:04:57.028 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-19T22:04:57.028 INFO:teuthology.orchestra.run.smithi176.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-19T22:04:57.028 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-19T22:04:57.028 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:04:57.028 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:04:57.028 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:04:57.028 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-02-19T22:04:57.028 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:04:57.029 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 53 M 2024-02-19T22:04:57.029 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 229 M 2024-02-19T22:04:57.029 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:04:57.059 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:04:57.059 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /sys 2024-02-19T22:04:57.059 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /proc 2024-02-19T22:04:57.059 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /dev 2024-02-19T22:04:57.059 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /run 2024-02-19T22:04:57.059 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /mnt 2024-02-19T22:04:57.059 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /var/tmp 2024-02-19T22:04:57.059 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /home 2024-02-19T22:04:57.059 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /tmp 2024-02-19T22:04:57.060 INFO:teuthology.orchestra.run.smithi184.stdout:skipping the directory /dev 2024-02-19T22:04:57.060 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:04:57.120 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 169 kB 00:00 2024-02-19T22:04:57.287 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 936 kB/s | 203 kB 00:00 2024-02-19T22:04:57.313 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 980 kB/s | 188 kB 00:00 2024-02-19T22:04:57.363 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-02-19T22:04:57.855 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): ceph-test-16.2.4-0.el8.x86_64.rpm 66 MB/s | 52 MB 00:00 2024-02-19T22:04:57.880 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 545 kB/s | 323 kB 00:00 2024-02-19T22:04:57.882 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:57.882 INFO:teuthology.orchestra.run.smithi176.stdout:Total 62 MB/s | 53 MB 00:00 2024-02-19T22:04:57.924 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:04:57.945 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:04:57.945 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:04:58.260 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:04:58.263 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:04:58.369 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:04:58.446 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:04:58.446 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:04:58.703 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:04:58.703 INFO:teuthology.orchestra.run.smithi184.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:04:58.768 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:04:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-19T22:04:59.191 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-19T22:04:59.329 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:04:59.409 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:04:59.412 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:04:59.412 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-19T22:04:59.412 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-19T22:04:59.412 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-19T22:04:59.413 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:04:59.413 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:04:59.413 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:04:59.413 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-19T22:04:59.413 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-19T22:04:59.413 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:04:59.413 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-19T22:04:59.413 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-19T22:04:59.413 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:04:59.413 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:04:59.413 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-19T22:04:59.413 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:04:59.413 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-19T22:04:59.413 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-19T22:04:59.414 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-19T22:04:59.414 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-19T22:04:59.414 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-19T22:04:59.414 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-19T22:04:59.414 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-19T22:04:59.414 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-19T22:04:59.414 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-19T22:04:59.414 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-19T22:04:59.414 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:04:59.414 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-19T22:04:59.414 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-19T22:04:59.553 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-19T22:04:59.679 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:05:00.393 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:05:00.394 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:00.394 INFO:teuthology.orchestra.run.smithi184.stdout:Upgraded: 2024-02-19T22:05:00.394 INFO:teuthology.orchestra.run.smithi184.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:00.394 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:05:00.394 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:00.394 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:00.394 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:00.394 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:00.394 INFO:teuthology.orchestra.run.smithi184.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-19T22:05:00.394 INFO:teuthology.orchestra.run.smithi184.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-19T22:05:00.394 INFO:teuthology.orchestra.run.smithi184.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-19T22:05:00.394 INFO:teuthology.orchestra.run.smithi184.stdout: leveldb-1.22-1.el8.x86_64 2024-02-19T22:05:00.394 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:00.395 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:00.396 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:00.396 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:00.396 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:05:00.697 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-test 2024-02-19T22:05:01.448 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:43 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:05:03.016 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:05:03.017 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:03.017 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-02-19T22:05:03.017 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:03.017 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:05:03.017 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-19T22:05:03.017 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-02-19T22:05:03.018 INFO:teuthology.orchestra.run.smithi184.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-19T22:05:03.018 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-19T22:05:03.018 INFO:teuthology.orchestra.run.smithi184.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-19T22:05:03.018 INFO:teuthology.orchestra.run.smithi184.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-19T22:05:03.018 INFO:teuthology.orchestra.run.smithi184.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-19T22:05:03.018 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:03.018 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:05:03.018 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:03.018 INFO:teuthology.orchestra.run.smithi184.stdout:Install 6 Packages 2024-02-19T22:05:03.018 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:03.019 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 53 M 2024-02-19T22:05:03.019 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 229 M 2024-02-19T22:05:03.019 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:05:03.162 INFO:teuthology.orchestra.run.smithi184.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 169 kB 00:00 2024-02-19T22:05:03.321 INFO:teuthology.orchestra.run.smithi184.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-02-19T22:05:03.379 INFO:teuthology.orchestra.run.smithi184.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2024-02-19T22:05:03.405 INFO:teuthology.orchestra.run.smithi184.stdout:(4/6): jq-1.6-8.el8.x86_64.rpm 688 kB/s | 203 kB 00:00 2024-02-19T22:05:03.905 INFO:teuthology.orchestra.run.smithi184.stdout:(5/6): ceph-test-16.2.4-0.el8.x86_64.rpm 65 MB/s | 52 MB 00:00 2024-02-19T22:05:03.930 INFO:teuthology.orchestra.run.smithi184.stdout:(6/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 128 kB/s | 71 kB 00:00 2024-02-19T22:05:03.932 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:03.933 INFO:teuthology.orchestra.run.smithi184.stdout:Total 58 MB/s | 53 MB 00:00 2024-02-19T22:05:03.975 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:05:03.997 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:05:03.997 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:05:04.306 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:05:04.309 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:05:04.565 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:05:04.785 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:05:04.828 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:05:05.058 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:05:05.058 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-19T22:05:05.059 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-19T22:05:05.059 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-19T22:05:05.059 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-19T22:05:05.059 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-19T22:05:05.086 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-19T22:05:05.275 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-19T22:05:05.425 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:05:05.470 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-19T22:05:05.470 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:05.470 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:05:05.470 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-19T22:05:05.470 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-19T22:05:05.470 INFO:teuthology.orchestra.run.smithi176.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-19T22:05:05.470 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:05.470 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:05:05.501 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:05:05.694 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph 2024-02-19T22:05:05.715 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-19T22:05:05.866 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:05:06.468 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:53 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:05:08.037 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:05:08.040 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:05:08.041 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-02-19T22:05:08.041 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:05:08.041 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:05:08.041 INFO:teuthology.orchestra.run.smithi176.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-19T22:05:08.041 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-02-19T22:05:08.041 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-19T22:05:08.041 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-19T22:05:08.041 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-19T22:05:08.042 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-19T22:05:08.042 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-19T22:05:08.042 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-19T22:05:08.042 INFO:teuthology.orchestra.run.smithi176.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-19T22:05:08.042 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-19T22:05:08.042 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-19T22:05:08.042 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-19T22:05:08.042 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-19T22:05:08.043 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-19T22:05:08.043 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-19T22:05:08.043 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-19T22:05:08.043 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-19T22:05:08.043 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-19T22:05:08.043 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-19T22:05:08.043 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-19T22:05:08.043 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-19T22:05:08.043 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-19T22:05:08.043 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-19T22:05:08.044 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-19T22:05:08.044 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-19T22:05:08.044 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-19T22:05:08.044 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-19T22:05:08.044 INFO:teuthology.orchestra.run.smithi176.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-19T22:05:08.044 INFO:teuthology.orchestra.run.smithi176.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-19T22:05:08.044 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-19T22:05:08.044 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-19T22:05:08.044 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-19T22:05:08.044 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-19T22:05:08.045 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:08.045 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:05:08.045 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:05:08.045 INFO:teuthology.orchestra.run.smithi176.stdout:Install 32 Packages 2024-02-19T22:05:08.045 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:08.046 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 31 M 2024-02-19T22:05:08.046 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 111 M 2024-02-19T22:05:08.046 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:05:08.606 INFO:teuthology.orchestra.run.smithi176.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 125 kB/s | 6.3 kB 00:00 2024-02-19T22:05:08.657 INFO:teuthology.orchestra.run.smithi176.stdout:(2/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 24 MB/s | 2.4 MB 00:00 2024-02-19T22:05:08.715 INFO:teuthology.orchestra.run.smithi176.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 9.9 MB/s | 1.6 MB 00:00 2024-02-19T22:05:08.749 INFO:teuthology.orchestra.run.smithi176.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-02-19T22:05:08.833 INFO:teuthology.orchestra.run.smithi176.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2024-02-19T22:05:09.066 INFO:teuthology.orchestra.run.smithi176.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2024-02-19T22:05:09.083 INFO:teuthology.orchestra.run.smithi176.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 155 kB/s | 39 kB 00:00 2024-02-19T22:05:09.108 INFO:teuthology.orchestra.run.smithi176.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 437 kB/s | 157 kB 00:00 2024-02-19T22:05:09.200 INFO:teuthology.orchestra.run.smithi176.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 487 kB/s | 57 kB 00:00 2024-02-19T22:05:09.217 INFO:teuthology.orchestra.run.smithi176.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 687 kB/s | 103 kB 00:00 2024-02-19T22:05:09.451 INFO:teuthology.orchestra.run.smithi176.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.3 MB/s | 457 kB 00:00 2024-02-19T22:05:09.518 INFO:teuthology.orchestra.run.smithi176.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 784 kB/s | 249 kB 00:00 2024-02-19T22:05:09.610 INFO:teuthology.orchestra.run.smithi176.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 1.6 MB/s | 641 kB 00:00 2024-02-19T22:05:09.668 INFO:teuthology.orchestra.run.smithi176.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 290 kB/s | 44 kB 00:00 2024-02-19T22:05:09.719 INFO:teuthology.orchestra.run.smithi176.stdout:(15/32): python3-pycparser-2.14-14.el8.noarch.r 407 kB/s | 109 kB 00:00 2024-02-19T22:05:09.786 INFO:teuthology.orchestra.run.smithi176.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.4 MB/s | 173 kB 00:00 2024-02-19T22:05:09.819 INFO:teuthology.orchestra.run.smithi176.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 886 kB/s | 185 kB 00:00 2024-02-19T22:05:09.845 INFO:teuthology.orchestra.run.smithi176.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 190 kB/s | 11 kB 00:00 2024-02-19T22:05:09.878 INFO:teuthology.orchestra.run.smithi176.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 300 kB/s | 18 kB 00:00 2024-02-19T22:05:09.904 INFO:teuthology.orchestra.run.smithi176.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 831 kB/s | 49 kB 00:00 2024-02-19T22:05:09.938 INFO:teuthology.orchestra.run.smithi176.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2024-02-19T22:05:09.987 INFO:teuthology.orchestra.run.smithi176.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2024-02-19T22:05:10.013 INFO:teuthology.orchestra.run.smithi176.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 207 kB/s | 16 kB 00:00 2024-02-19T22:05:10.038 INFO:teuthology.orchestra.run.smithi176.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2024-02-19T22:05:10.063 INFO:teuthology.orchestra.run.smithi176.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2024-02-19T22:05:10.089 INFO:teuthology.orchestra.run.smithi176.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2024-02-19T22:05:10.113 INFO:teuthology.orchestra.run.smithi176.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2024-02-19T22:05:10.139 INFO:teuthology.orchestra.run.smithi176.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 364 kB/s | 27 kB 00:00 2024-02-19T22:05:10.164 INFO:teuthology.orchestra.run.smithi176.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2024-02-19T22:05:10.189 INFO:teuthology.orchestra.run.smithi176.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.2 MB/s | 251 kB 00:00 2024-02-19T22:05:10.215 INFO:teuthology.orchestra.run.smithi176.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2024-02-19T22:05:10.240 INFO:teuthology.orchestra.run.smithi176.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2024-02-19T22:05:10.241 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:10.241 INFO:teuthology.orchestra.run.smithi176.stdout:Total 14 MB/s | 31 MB 00:02 2024-02-19T22:05:10.323 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:05:10.362 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:05:10.362 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:05:10.662 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:05:10.665 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:05:10.699 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:05:10.939 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:05:11.225 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:05:11.225 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-19T22:05:11.226 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-19T22:05:11.226 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-19T22:05:11.226 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-19T22:05:11.226 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-19T22:05:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-19T22:05:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:11.655 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:05:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-19T22:05:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-19T22:05:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-19T22:05:11.655 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:11.655 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:05:11.766 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:05:11.886 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph 2024-02-19T22:05:12.007 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-19T22:05:12.193 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-19T22:05:12.427 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-19T22:05:12.649 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:00:54 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:05:12.674 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-19T22:05:12.856 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-19T22:05:13.036 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-19T22:05:13.213 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-19T22:05:13.415 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-19T22:05:13.625 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-19T22:05:13.837 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-19T22:05:14.067 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-19T22:05:14.184 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:05:14.187 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:14.187 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-02-19T22:05:14.187 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:14.187 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:05:14.187 INFO:teuthology.orchestra.run.smithi184.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-19T22:05:14.187 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-02-19T22:05:14.187 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-19T22:05:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-19T22:05:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-19T22:05:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-19T22:05:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-19T22:05:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-19T22:05:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-19T22:05:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-19T22:05:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-19T22:05:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-19T22:05:14.188 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-19T22:05:14.189 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-19T22:05:14.189 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-19T22:05:14.189 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-19T22:05:14.189 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-19T22:05:14.189 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-19T22:05:14.189 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-19T22:05:14.189 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-19T22:05:14.189 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-19T22:05:14.189 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-19T22:05:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-19T22:05:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-19T22:05:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-19T22:05:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-19T22:05:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-19T22:05:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-19T22:05:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-19T22:05:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-19T22:05:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-19T22:05:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-19T22:05:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-19T22:05:14.190 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:14.190 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:05:14.191 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:14.191 INFO:teuthology.orchestra.run.smithi184.stdout:Install 32 Packages 2024-02-19T22:05:14.191 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:14.191 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 31 M 2024-02-19T22:05:14.191 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 111 M 2024-02-19T22:05:14.192 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:05:14.290 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-19T22:05:14.598 INFO:teuthology.orchestra.run.smithi184.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 138 kB/s | 6.3 kB 00:00 2024-02-19T22:05:14.643 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-19T22:05:14.648 INFO:teuthology.orchestra.run.smithi184.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-02-19T22:05:14.691 INFO:teuthology.orchestra.run.smithi184.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 17 MB/s | 2.4 MB 00:00 2024-02-19T22:05:14.716 INFO:teuthology.orchestra.run.smithi184.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-02-19T22:05:14.833 INFO:teuthology.orchestra.run.smithi184.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-02-19T22:05:14.887 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-19T22:05:15.058 INFO:teuthology.orchestra.run.smithi184.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2024-02-19T22:05:15.067 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-19T22:05:15.075 INFO:teuthology.orchestra.run.smithi184.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 160 kB/s | 39 kB 00:00 2024-02-19T22:05:15.100 INFO:teuthology.orchestra.run.smithi184.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 409 kB/s | 157 kB 00:00 2024-02-19T22:05:15.134 INFO:teuthology.orchestra.run.smithi184.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 971 kB/s | 57 kB 00:00 2024-02-19T22:05:15.234 INFO:teuthology.orchestra.run.smithi184.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 590 kB/s | 103 kB 00:00 2024-02-19T22:05:15.293 INFO:teuthology.orchestra.run.smithi184.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.3 MB/s | 457 kB 00:00 2024-02-19T22:05:15.327 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-19T22:05:15.435 INFO:teuthology.orchestra.run.smithi184.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 828 kB/s | 249 kB 00:00 2024-02-19T22:05:15.485 INFO:teuthology.orchestra.run.smithi184.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 873 kB/s | 44 kB 00:00 2024-02-19T22:05:15.519 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:05:15.535 INFO:teuthology.orchestra.run.smithi184.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 3.6 MB/s | 185 kB 00:00 2024-02-19T22:05:15.560 INFO:teuthology.orchestra.run.smithi184.stdout:(15/32): python3-pycparser-2.14-14.el8.noarch.r 408 kB/s | 109 kB 00:00 2024-02-19T22:05:15.586 INFO:teuthology.orchestra.run.smithi184.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 3.4 MB/s | 173 kB 00:00 2024-02-19T22:05:15.611 INFO:teuthology.orchestra.run.smithi184.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 448 kB/s | 11 kB 00:00 2024-02-19T22:05:15.645 INFO:teuthology.orchestra.run.smithi184.stdout:(18/32): python3-cryptography-3.2.1-7.el8.x86_6 1.5 MB/s | 641 kB 00:00 2024-02-19T22:05:15.662 INFO:teuthology.orchestra.run.smithi184.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 349 kB/s | 18 kB 00:00 2024-02-19T22:05:15.687 INFO:teuthology.orchestra.run.smithi184.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 2.3 MB/s | 59 kB 00:00 2024-02-19T22:05:15.712 INFO:teuthology.orchestra.run.smithi184.stdout:(21/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.5 MB/s | 384 kB 00:00 2024-02-19T22:05:15.729 INFO:teuthology.orchestra.run.smithi184.stdout:(22/32): python3-logutils-0.3.5-11.el8.noarch.r 578 kB/s | 49 kB 00:00 2024-02-19T22:05:15.746 INFO:teuthology.orchestra.run.smithi184.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 464 kB/s | 16 kB 00:00 2024-02-19T22:05:15.763 INFO:teuthology.orchestra.run.smithi184.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 562 kB/s | 19 kB 00:00 2024-02-19T22:05:15.780 INFO:teuthology.orchestra.run.smithi184.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 714 kB/s | 24 kB 00:00 2024-02-19T22:05:15.797 INFO:teuthology.orchestra.run.smithi184.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 870 kB/s | 29 kB 00:00 2024-02-19T22:05:15.814 INFO:teuthology.orchestra.run.smithi184.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 822 kB/s | 27 kB 00:00 2024-02-19T22:05:15.829 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-19T22:05:15.839 INFO:teuthology.orchestra.run.smithi184.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2024-02-19T22:05:15.865 INFO:teuthology.orchestra.run.smithi184.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.7 MB/s | 255 kB 00:00 2024-02-19T22:05:15.881 INFO:teuthology.orchestra.run.smithi184.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 2.0 MB/s | 85 kB 00:00 2024-02-19T22:05:15.907 INFO:teuthology.orchestra.run.smithi184.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.7 MB/s | 251 kB 00:00 2024-02-19T22:05:15.963 INFO:teuthology.orchestra.run.smithi184.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 235 kB/s | 23 kB 00:00 2024-02-19T22:05:15.964 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:15.964 INFO:teuthology.orchestra.run.smithi184.stdout:Total 17 MB/s | 31 MB 00:01 2024-02-19T22:05:16.044 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:05:16.048 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-19T22:05:16.082 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:05:16.082 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:05:16.236 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-19T22:05:16.415 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:05:16.418 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:05:16.423 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-19T22:05:16.693 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-19T22:05:16.896 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-19T22:05:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-19T22:05:17.469 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-19T22:05:17.534 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:05:17.765 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-19T22:05:17.778 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-19T22:05:17.970 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-19T22:05:18.189 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-19T22:05:18.197 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-19T22:05:18.368 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:05:18.420 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-19T22:05:18.568 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-19T22:05:18.723 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-19T22:05:18.892 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-19T22:05:19.036 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-19T22:05:19.210 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-19T22:05:19.389 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-19T22:05:19.568 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-19T22:05:19.791 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-19T22:05:19.914 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:05:20.126 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:05:20.175 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-19T22:05:20.425 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-19T22:05:20.561 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:05:20.613 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-19T22:05:20.793 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:05:20.828 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-19T22:05:20.998 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:05:21.014 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:05:21.276 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:05:21.293 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:05:21.306 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-19T22:05:21.450 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:05:21.502 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-19T22:05:21.666 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-19T22:05:21.871 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:05:21.871 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-19T22:05:21.871 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-19T22:05:21.871 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-19T22:05:21.871 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-19T22:05:21.871 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-19T22:05:21.871 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-19T22:05:21.871 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-19T22:05:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-19T22:05:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-19T22:05:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-19T22:05:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-19T22:05:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-19T22:05:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-19T22:05:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-19T22:05:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-19T22:05:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-19T22:05:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:05:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-19T22:05:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-19T22:05:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-19T22:05:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-19T22:05:21.873 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-19T22:05:21.873 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-19T22:05:21.873 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-19T22:05:21.873 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-19T22:05:21.873 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-19T22:05:21.873 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-19T22:05:21.873 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-19T22:05:21.873 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-19T22:05:21.873 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-19T22:05:21.873 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-19T22:05:21.893 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-19T22:05:22.110 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-19T22:05:22.325 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-19T22:05:22.559 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-19T22:05:22.799 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-19T22:05:22.799 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:22.799 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:05:22.799 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:22.799 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:22.799 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:22.799 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-19T22:05:22.799 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:22.799 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:22.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-19T22:05:22.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-19T22:05:22.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-19T22:05:22.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-19T22:05:22.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-19T22:05:22.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-19T22:05:22.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-19T22:05:22.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-19T22:05:22.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-19T22:05:22.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-19T22:05:22.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-19T22:05:22.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-19T22:05:22.800 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-19T22:05:22.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend-2.6-1.el8.noarch 2024-02-19T22:05:22.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-19T22:05:22.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-19T22:05:22.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-19T22:05:22.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-19T22:05:22.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-19T22:05:22.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-19T22:05:22.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-19T22:05:22.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-19T22:05:22.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-19T22:05:22.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-19T22:05:22.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-19T22:05:22.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-19T22:05:22.802 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:22.802 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:05:22.891 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-19T22:05:23.092 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-base 2024-02-19T22:05:23.170 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-19T22:05:23.536 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-19T22:05:23.740 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:05:23.850 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:10 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:05:25.281 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:05:25.369 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:05:25.417 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:05:25.418 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-02-19T22:05:25.418 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:05:25.498 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:05:25.532 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install cephadm 2024-02-19T22:05:25.931 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:05:26.148 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:05:26.283 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:13 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:05:26.370 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:05:26.555 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:05:26.572 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:05:26.721 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:05:27.116 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:05:27.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-19T22:05:27.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-19T22:05:27.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-19T22:05:27.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-19T22:05:27.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-19T22:05:27.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-19T22:05:27.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-19T22:05:27.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-19T22:05:27.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-19T22:05:27.116 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-19T22:05:27.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-19T22:05:27.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-19T22:05:27.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-19T22:05:27.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-19T22:05:27.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-19T22:05:27.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-19T22:05:27.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:05:27.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-19T22:05:27.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-19T22:05:27.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-19T22:05:27.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-19T22:05:27.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-19T22:05:27.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-19T22:05:27.117 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-19T22:05:27.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-19T22:05:27.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-19T22:05:27.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-19T22:05:27.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-19T22:05:27.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-19T22:05:27.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-19T22:05:27.118 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-19T22:05:27.831 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:05:27.831 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:05:27.831 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-02-19T22:05:27.831 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:05:27.832 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:05:27.832 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-19T22:05:27.832 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:27.832 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:05:27.832 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:05:27.832 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-02-19T22:05:27.832 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:27.833 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 71 k 2024-02-19T22:05:27.833 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 290 k 2024-02-19T22:05:27.833 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:05:27.878 INFO:teuthology.orchestra.run.smithi176.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2024-02-19T22:05:27.879 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:27.879 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.5 MB/s | 71 kB 00:00 2024-02-19T22:05:27.879 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:05:27.886 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:05:27.886 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:05:27.888 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:05:27.891 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:05:27.930 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:05:28.040 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-19T22:05:28.040 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:28.040 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:05:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-19T22:05:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-19T22:05:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-19T22:05:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-19T22:05:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-19T22:05:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-19T22:05:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-19T22:05:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-19T22:05:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-19T22:05:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-19T22:05:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-19T22:05:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-19T22:05:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-19T22:05:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-19T22:05:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: python3-portend-2.6-1.el8.noarch 2024-02-19T22:05:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-19T22:05:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-19T22:05:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-19T22:05:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-19T22:05:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-19T22:05:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-19T22:05:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-19T22:05:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-19T22:05:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-19T22:05:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-19T22:05:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-19T22:05:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-19T22:05:28.044 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:28.044 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:05:28.323 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-base 2024-02-19T22:05:28.408 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:05:28.563 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:05:28.786 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:05:29.085 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:11 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:05:29.162 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:05:29.162 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:29.163 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:05:29.163 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-19T22:05:29.163 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:29.163 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:05:29.377 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T22:05:30.136 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:17 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:05:30.592 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:05:30.639 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:05:30.640 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-02-19T22:05:30.640 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:05:30.755 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install cephadm 2024-02-19T22:05:31.501 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:13 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:05:31.686 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:05:31.687 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:05:31.687 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-02-19T22:05:31.687 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:05:31.688 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:05:31.688 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-19T22:05:31.688 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:31.688 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:05:31.688 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:05:31.688 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-02-19T22:05:31.688 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:31.688 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 163 k 2024-02-19T22:05:31.688 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 587 k 2024-02-19T22:05:31.688 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:05:31.724 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 4.4 MB/s | 163 kB 00:00 2024-02-19T22:05:31.725 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:31.725 INFO:teuthology.orchestra.run.smithi176.stdout:Total 4.3 MB/s | 163 kB 00:00 2024-02-19T22:05:31.725 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:05:31.736 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:05:31.736 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:05:31.783 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:05:31.786 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:05:31.897 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:05:33.045 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:05:33.046 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:33.046 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-02-19T22:05:33.046 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:33.046 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:05:33.046 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-19T22:05:33.046 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:33.047 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:05:33.047 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:33.047 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-02-19T22:05:33.047 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:33.047 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 71 k 2024-02-19T22:05:33.047 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 290 k 2024-02-19T22:05:33.047 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:05:33.086 INFO:teuthology.orchestra.run.smithi184.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 71 kB 00:00 2024-02-19T22:05:33.087 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:33.087 INFO:teuthology.orchestra.run.smithi184.stdout:Total 1.7 MB/s | 71 kB 00:00 2024-02-19T22:05:33.087 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:05:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:05:33.094 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:05:33.096 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:05:33.098 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:05:33.151 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:05:33.599 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:33.657 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:05:33.866 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:05:34.036 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:34.095 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:05:34.402 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:34.402 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:34.402 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:05:34.403 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:34.403 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:34.403 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:05:34.470 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:05:34.470 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:34.470 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:05:34.470 INFO:teuthology.orchestra.run.smithi184.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-19T22:05:34.470 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:34.470 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:05:34.630 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr 2024-02-19T22:05:34.704 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T22:05:35.389 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:22 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:05:35.467 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:17 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:05:36.888 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:05:36.934 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:05:36.936 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-02-19T22:05:36.936 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:05:37.022 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:05:37.023 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:37.023 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repo Size 2024-02-19T22:05:37.023 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:37.023 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:05:37.023 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-19T22:05:37.023 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:37.023 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:05:37.023 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:37.023 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-02-19T22:05:37.023 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:37.023 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 163 k 2024-02-19T22:05:37.023 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 587 k 2024-02-19T22:05:37.023 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:05:37.057 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T22:05:37.074 INFO:teuthology.orchestra.run.smithi184.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.2 MB/s | 163 kB 00:00 2024-02-19T22:05:37.075 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:37.075 INFO:teuthology.orchestra.run.smithi184.stdout:Total 3.1 MB/s | 163 kB 00:00 2024-02-19T22:05:37.075 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:05:37.085 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:05:37.085 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:05:37.132 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:05:37.136 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:05:37.249 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:05:37.551 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:37.814 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:24 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:05:38.972 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:39.136 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:39.136 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:39.136 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:05:39.137 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:39.137 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:39.137 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:05:39.372 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:05:39.376 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:05:39.376 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-02-19T22:05:39.377 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:05:39.377 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:05:39.377 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-19T22:05:39.377 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-02-19T22:05:39.377 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-19T22:05:39.378 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-19T22:05:39.378 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-19T22:05:39.378 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-19T22:05:39.378 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-19T22:05:39.378 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:39.379 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:05:39.379 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:05:39.379 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-02-19T22:05:39.379 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:39.379 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 3.0 M 2024-02-19T22:05:39.379 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 15 M 2024-02-19T22:05:39.379 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:05:39.388 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr 2024-02-19T22:05:39.770 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 421 kB/s | 24 kB 00:00 2024-02-19T22:05:39.795 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 108 kB/s | 8.9 kB 00:00 2024-02-19T22:05:39.862 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 18 MB/s | 2.7 MB 00:00 2024-02-19T22:05:40.029 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 164 kB/s | 43 kB 00:00 2024-02-19T22:05:40.046 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 134 kB/s | 34 kB 00:00 2024-02-19T22:05:40.155 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:22 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:05:40.204 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 582 kB/s | 196 kB 00:00 2024-02-19T22:05:40.207 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:40.208 INFO:teuthology.orchestra.run.smithi176.stdout:Total 3.5 MB/s | 3.0 MB 00:00 2024-02-19T22:05:40.219 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:05:40.230 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:05:40.230 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:05:40.392 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:05:40.396 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:05:40.706 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:05:40.911 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-19T22:05:41.165 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-19T22:05:41.408 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-19T22:05:41.528 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-19T22:05:41.653 INFO:teuthology.orchestra.run.smithi184.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:05:41.703 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:05:41.703 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-02-19T22:05:41.704 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:05:41.811 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T22:05:42.486 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-19T22:05:42.562 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:24 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:05:42.755 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:05:43.039 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:05:43.039 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:05:43.039 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-19T22:05:43.040 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-19T22:05:43.040 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-19T22:05:43.040 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-19T22:05:43.445 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-19T22:05:43.445 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:43.446 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:05:43.446 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-19T22:05:43.446 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-19T22:05:43.446 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-19T22:05:43.446 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-19T22:05:43.446 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-19T22:05:43.446 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-19T22:05:43.446 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:43.446 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:05:43.575 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T22:05:44.108 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:05:44.109 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:44.109 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-02-19T22:05:44.109 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:44.109 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:05:44.109 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-19T22:05:44.109 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-02-19T22:05:44.109 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-19T22:05:44.109 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-19T22:05:44.109 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-19T22:05:44.109 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-19T22:05:44.110 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-19T22:05:44.110 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:44.110 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:05:44.110 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:05:44.110 INFO:teuthology.orchestra.run.smithi184.stdout:Install 6 Packages 2024-02-19T22:05:44.110 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:44.110 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 3.0 M 2024-02-19T22:05:44.110 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 15 M 2024-02-19T22:05:44.110 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:05:44.335 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:31 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:05:44.411 INFO:teuthology.orchestra.run.smithi184.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 107 kB/s | 24 kB 00:00 2024-02-19T22:05:44.436 INFO:teuthology.orchestra.run.smithi184.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 36 kB/s | 8.9 kB 00:00 2024-02-19T22:05:44.503 INFO:teuthology.orchestra.run.smithi184.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 8.4 MB/s | 2.7 MB 00:00 2024-02-19T22:05:44.662 INFO:teuthology.orchestra.run.smithi184.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 170 kB/s | 43 kB 00:00 2024-02-19T22:05:44.954 INFO:teuthology.orchestra.run.smithi184.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 65 kB/s | 34 kB 00:00 2024-02-19T22:05:45.896 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:05:45.897 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-02-19T22:05:45.897 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-02-19T22:05:45.897 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-02-19T22:05:45.898 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:05:45.898 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-19T22:05:45.898 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-02-19T22:05:45.898 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-19T22:05:45.898 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-19T22:05:45.898 INFO:teuthology.orchestra.run.smithi176.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-19T22:05:45.898 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-19T22:05:45.898 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-19T22:05:45.898 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-19T22:05:45.899 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-19T22:05:45.899 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:45.899 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:05:45.899 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-02-19T22:05:45.899 INFO:teuthology.orchestra.run.smithi176.stdout:Install 8 Packages 2024-02-19T22:05:45.899 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:45.899 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 37 M 2024-02-19T22:05:45.899 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 210 M 2024-02-19T22:05:45.899 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:05:46.284 INFO:teuthology.orchestra.run.smithi176.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 56 MB/s | 8.6 MB 00:00 2024-02-19T22:05:46.637 INFO:teuthology.orchestra.run.smithi184.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 92 kB/s | 196 kB 00:02 2024-02-19T22:05:46.639 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:46.639 INFO:teuthology.orchestra.run.smithi184.stdout:Total 1.2 MB/s | 3.0 MB 00:02 2024-02-19T22:05:46.651 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:05:46.661 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:05:46.661 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:05:46.676 INFO:teuthology.orchestra.run.smithi176.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 8.7 MB/s | 4.7 MB 00:00 2024-02-19T22:05:46.727 INFO:teuthology.orchestra.run.smithi176.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 4.4 MB/s | 225 kB 00:00 2024-02-19T22:05:46.802 INFO:teuthology.orchestra.run.smithi176.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 7.2 MB/s | 4.9 MB 00:00 2024-02-19T22:05:46.842 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:05:46.845 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:05:47.086 INFO:teuthology.orchestra.run.smithi176.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.7 MB/s | 3.7 MB 00:00 2024-02-19T22:05:47.107 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:05:47.236 INFO:teuthology.orchestra.run.smithi176.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.5 MB/s | 645 kB 00:00 2024-02-19T22:05:47.344 INFO:teuthology.orchestra.run.smithi176.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 667 kB/s | 172 kB 00:00 2024-02-19T22:05:47.365 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-19T22:05:47.577 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-19T22:05:47.744 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-19T22:05:47.890 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-19T22:05:48.844 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-19T22:05:48.853 INFO:teuthology.orchestra.run.smithi176.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 6.7 MB/s | 14 MB 00:02 2024-02-19T22:05:48.853 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:48.854 INFO:teuthology.orchestra.run.smithi176.stdout:Total 13 MB/s | 37 MB 00:02 2024-02-19T22:05:49.086 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:05:49.104 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:05:49.104 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:05:49.127 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:05:49.401 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:05:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:05:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-19T22:05:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-19T22:05:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-19T22:05:49.402 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-19T22:05:49.417 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:05:49.420 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:05:49.988 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-19T22:05:49.988 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:49.988 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:05:49.988 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-19T22:05:49.988 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-19T22:05:49.988 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-19T22:05:49.988 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-19T22:05:49.989 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-19T22:05:49.989 INFO:teuthology.orchestra.run.smithi184.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-19T22:05:49.989 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:49.989 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:05:50.134 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T22:05:50.601 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:05:50.754 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:05:50.888 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:05:50.898 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:32 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:05:51.090 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:05:51.695 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:05:51.870 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:05:52.436 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:05:52.438 INFO:teuthology.orchestra.run.smithi184.stdout:============================================================================================ 2024-02-19T22:05:52.438 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-02-19T22:05:52.438 INFO:teuthology.orchestra.run.smithi184.stdout:============================================================================================ 2024-02-19T22:05:52.438 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:05:52.438 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-19T22:05:52.438 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-02-19T22:05:52.438 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-19T22:05:52.438 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-19T22:05:52.438 INFO:teuthology.orchestra.run.smithi184.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-19T22:05:52.438 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-19T22:05:52.439 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-19T22:05:52.439 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-19T22:05:52.439 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-19T22:05:52.439 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:52.439 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:05:52.439 INFO:teuthology.orchestra.run.smithi184.stdout:============================================================================================ 2024-02-19T22:05:52.439 INFO:teuthology.orchestra.run.smithi184.stdout:Install 8 Packages 2024-02-19T22:05:52.439 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:05:52.440 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 37 M 2024-02-19T22:05:52.440 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 210 M 2024-02-19T22:05:52.440 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:05:52.487 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:05:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:05:52.666 INFO:teuthology.orchestra.run.smithi184.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 59 MB/s | 8.6 MB 00:00 2024-02-19T22:05:53.347 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:05:53.467 INFO:teuthology.orchestra.run.smithi184.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 5.0 MB/s | 4.7 MB 00:00 2024-02-19T22:05:53.592 INFO:teuthology.orchestra.run.smithi184.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.5 MB/s | 4.9 MB 00:01 2024-02-19T22:05:53.636 INFO:teuthology.orchestra.run.smithi184.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.3 MB/s | 225 kB 00:00 2024-02-19T22:05:53.707 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:05:53.768 INFO:teuthology.orchestra.run.smithi184.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.4 MB/s | 3.7 MB 00:01 2024-02-19T22:05:53.953 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:05:53.993 INFO:teuthology.orchestra.run.smithi184.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 645 kB 00:00 2024-02-19T22:05:54.018 INFO:teuthology.orchestra.run.smithi184.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 689 kB/s | 172 kB 00:00 2024-02-19T22:05:55.733 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:05:56.154 INFO:teuthology.orchestra.run.smithi184.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 5.5 MB/s | 14 MB 00:02 2024-02-19T22:05:56.154 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:56.155 INFO:teuthology.orchestra.run.smithi184.stdout:Total 10 MB/s | 37 MB 00:03 2024-02-19T22:05:56.385 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:05:56.404 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:05:56.404 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:05:56.742 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:05:56.745 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:05:57.020 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-19T22:05:57.200 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:05:57.354 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:05:57.499 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:05:57.500 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:05:57.501 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-19T22:05:57.501 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-19T22:05:57.501 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:05:57.501 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-19T22:05:57.501 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:05:57.501 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-19T22:05:57.501 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-19T22:05:57.645 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:05:57.847 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:05:57.917 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-19T22:05:57.918 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:57.918 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:05:57.918 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-19T22:05:57.918 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-19T22:05:57.918 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-19T22:05:57.918 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-19T22:05:57.918 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-19T22:05:57.918 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-19T22:05:57.918 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-19T22:05:57.918 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-19T22:05:57.918 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:05:57.918 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:05:58.065 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-rook 2024-02-19T22:05:58.445 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:05:58.591 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:05:58.824 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:45 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:05:59.205 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:05:59.364 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:06:00.070 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:06:00.366 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:00.367 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:00.367 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-02-19T22:06:00.368 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:00.368 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:06:00.368 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-19T22:06:00.368 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-02-19T22:06:00.368 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-19T22:06:00.368 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-19T22:06:00.368 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-19T22:06:00.368 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-19T22:06:00.368 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-19T22:06:00.368 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-19T22:06:00.368 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-19T22:06:00.368 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-19T22:06:00.369 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-19T22:06:00.369 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-19T22:06:00.369 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-19T22:06:00.369 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-19T22:06:00.369 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:00.369 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:06:00.369 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:00.369 INFO:teuthology.orchestra.run.smithi176.stdout:Install 13 Packages 2024-02-19T22:06:00.369 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:00.370 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 1.9 M 2024-02-19T22:06:00.370 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 24 M 2024-02-19T22:06:00.370 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:06:00.403 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:06:00.577 INFO:teuthology.orchestra.run.smithi176.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 425 kB/s | 27 kB 00:00 2024-02-19T22:06:00.711 INFO:teuthology.orchestra.run.smithi176.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 101 kB/s | 20 kB 00:00 2024-02-19T22:06:00.736 INFO:teuthology.orchestra.run.smithi176.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 120 kB/s | 27 kB 00:00 2024-02-19T22:06:00.872 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:06:00.995 INFO:teuthology.orchestra.run.smithi176.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 600 kB/s | 155 kB 00:00 2024-02-19T22:06:01.201 INFO:teuthology.orchestra.run.smithi176.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 207 kB/s | 43 kB 00:00 2024-02-19T22:06:01.387 INFO:teuthology.orchestra.run.smithi176.stdout:(6/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 156 kB/s | 126 kB 00:00 2024-02-19T22:06:01.487 INFO:teuthology.orchestra.run.smithi176.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 142 kB/s | 110 kB 00:00 2024-02-19T22:06:02.313 INFO:teuthology.orchestra.run.smithi176.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 20 kB/s | 16 kB 00:00 2024-02-19T22:06:02.371 INFO:teuthology.orchestra.run.smithi176.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 28 kB/s | 33 kB 00:01 2024-02-19T22:06:02.646 INFO:teuthology.orchestra.run.smithi176.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 70 kB/s | 81 kB 00:01 2024-02-19T22:06:02.653 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:06:02.822 INFO:teuthology.orchestra.run.smithi176.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 143 kB/s | 64 kB 00:00 2024-02-19T22:06:02.972 INFO:teuthology.orchestra.run.smithi176.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 188 kB/s | 61 kB 00:00 2024-02-19T22:06:03.631 INFO:teuthology.orchestra.run.smithi176.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 906 kB/s | 1.2 MB 00:01 2024-02-19T22:06:03.634 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:03.634 INFO:teuthology.orchestra.run.smithi176.stdout:Total 599 kB/s | 1.9 MB 00:03 2024-02-19T22:06:03.691 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:06:03.707 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:06:03.707 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:06:03.796 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:06:03.799 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:06:03.918 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-19T22:06:04.132 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:06:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:06:04.447 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:06:04.447 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-19T22:06:04.447 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-19T22:06:04.448 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:06:04.448 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-19T22:06:04.448 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:06:04.448 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-19T22:06:04.448 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-19T22:06:04.554 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-19T22:06:04.752 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-19T22:06:04.841 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-19T22:06:04.841 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:04.841 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:06:04.841 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-19T22:06:04.841 INFO:teuthology.orchestra.run.smithi184.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-19T22:06:04.841 INFO:teuthology.orchestra.run.smithi184.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-19T22:06:04.841 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-19T22:06:04.841 INFO:teuthology.orchestra.run.smithi184.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-19T22:06:04.841 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-19T22:06:04.841 INFO:teuthology.orchestra.run.smithi184.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-19T22:06:04.841 INFO:teuthology.orchestra.run.smithi184.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-19T22:06:04.841 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:04.842 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:04.944 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-rook 2024-02-19T22:06:04.962 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-19T22:06:05.183 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-19T22:06:05.374 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-19T22:06:05.606 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-19T22:06:05.701 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:47 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:06:05.924 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-19T22:06:06.148 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-19T22:06:07.166 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-19T22:06:07.273 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:06:07.275 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:07.275 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-02-19T22:06:07.275 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:07.275 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:06:07.275 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-19T22:06:07.275 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-02-19T22:06:07.275 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-19T22:06:07.275 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-19T22:06:07.275 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-19T22:06:07.275 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-19T22:06:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-19T22:06:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-19T22:06:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-19T22:06:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-19T22:06:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-19T22:06:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-19T22:06:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-19T22:06:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-19T22:06:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:07.276 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:06:07.276 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:07.276 INFO:teuthology.orchestra.run.smithi184.stdout:Install 13 Packages 2024-02-19T22:06:07.276 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:07.277 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 1.9 M 2024-02-19T22:06:07.277 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 24 M 2024-02-19T22:06:07.277 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:06:07.441 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-19T22:06:07.642 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-19T22:06:07.858 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-19T22:06:08.009 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:06:08.305 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:06:08.305 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-19T22:06:08.305 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-19T22:06:08.305 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-19T22:06:08.305 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-19T22:06:08.305 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-19T22:06:08.305 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-19T22:06:08.305 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-19T22:06:08.305 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-19T22:06:08.305 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-19T22:06:08.305 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-19T22:06:08.305 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-19T22:06:08.305 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-19T22:06:08.350 INFO:teuthology.orchestra.run.smithi184.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 540 kB/s | 27 kB 00:00 2024-02-19T22:06:08.417 INFO:teuthology.orchestra.run.smithi184.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 171 kB/s | 20 kB 00:00 2024-02-19T22:06:08.434 INFO:teuthology.orchestra.run.smithi184.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 199 kB/s | 27 kB 00:00 2024-02-19T22:06:08.518 INFO:teuthology.orchestra.run.smithi184.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 757 kB/s | 126 kB 00:00 2024-02-19T22:06:08.543 INFO:teuthology.orchestra.run.smithi184.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 876 kB/s | 110 kB 00:00 2024-02-19T22:06:08.702 INFO:teuthology.orchestra.run.smithi184.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 580 kB/s | 155 kB 00:00 2024-02-19T22:06:08.719 INFO:teuthology.orchestra.run.smithi184.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 212 kB/s | 43 kB 00:00 2024-02-19T22:06:08.801 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-19T22:06:08.801 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:08.801 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:06:08.801 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-19T22:06:08.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-19T22:06:08.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-19T22:06:08.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-19T22:06:08.802 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-19T22:06:08.802 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-19T22:06:08.802 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-19T22:06:08.802 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-19T22:06:08.802 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-19T22:06:08.802 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-19T22:06:08.802 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-19T22:06:08.802 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-19T22:06:08.802 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-19T22:06:08.802 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:08.802 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:08.932 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T22:06:08.952 INFO:teuthology.orchestra.run.smithi184.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 81 kB/s | 33 kB 00:00 2024-02-19T22:06:09.052 INFO:teuthology.orchestra.run.smithi184.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 46 kB/s | 16 kB 00:00 2024-02-19T22:06:09.211 INFO:teuthology.orchestra.run.smithi184.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 165 kB/s | 81 kB 00:00 2024-02-19T22:06:09.295 INFO:teuthology.orchestra.run.smithi184.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 265 kB/s | 64 kB 00:00 2024-02-19T22:06:09.357 INFO:teuthology.orchestra.run.smithi184.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 431 kB/s | 61 kB 00:00 2024-02-19T22:06:09.504 INFO:teuthology.orchestra.run.smithi184.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.1 MB/s | 1.2 MB 00:00 2024-02-19T22:06:09.510 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:09.510 INFO:teuthology.orchestra.run.smithi184.stdout:Total 877 kB/s | 1.9 MB 00:02 2024-02-19T22:06:09.567 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:06:09.583 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:06:09.583 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:06:09.671 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:06:09.675 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:06:09.681 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:56 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:06:10.201 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:06:10.408 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-19T22:06:10.579 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-19T22:06:10.807 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-19T22:06:11.045 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-19T22:06:11.227 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:11.228 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:11.228 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-02-19T22:06:11.228 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:11.228 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:06:11.228 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-19T22:06:11.229 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-02-19T22:06:11.229 INFO:teuthology.orchestra.run.smithi176.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-19T22:06:11.229 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-19T22:06:11.229 INFO:teuthology.orchestra.run.smithi176.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-19T22:06:11.229 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-19T22:06:11.229 INFO:teuthology.orchestra.run.smithi176.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-19T22:06:11.229 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:11.229 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:06:11.229 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:11.229 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-02-19T22:06:11.229 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:11.230 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 5.9 M 2024-02-19T22:06:11.230 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 24 M 2024-02-19T22:06:11.230 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:06:11.268 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-19T22:06:11.508 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-19T22:06:11.825 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-19T22:06:12.042 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-19T22:06:12.199 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.2 MB/s | 89 kB 00:00 2024-02-19T22:06:12.322 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 173 kB/s | 21 kB 00:00 2024-02-19T22:06:12.456 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.6 MB/s | 355 kB 00:00 2024-02-19T22:06:12.498 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 1.2 MB/s | 49 kB 00:00 2024-02-19T22:06:12.781 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.3 MB/s | 4.8 MB 00:00 2024-02-19T22:06:12.998 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 664 kB/s | 578 kB 00:00 2024-02-19T22:06:13.000 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:13.000 INFO:teuthology.orchestra.run.smithi176.stdout:Total 3.3 MB/s | 5.9 MB 00:01 2024-02-19T22:06:13.054 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:06:13.064 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:06:13.064 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:06:13.085 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-19T22:06:13.165 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:06:13.168 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:06:13.370 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-19T22:06:13.451 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:06:13.569 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-19T22:06:13.708 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-19T22:06:13.746 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-19T22:06:13.946 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:06:13.995 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-19T22:06:14.258 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:06:14.258 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-19T22:06:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-19T22:06:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-19T22:06:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-19T22:06:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-19T22:06:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-19T22:06:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-19T22:06:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-19T22:06:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-19T22:06:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-19T22:06:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-19T22:06:14.259 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-19T22:06:14.712 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-19T22:06:14.712 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:14.712 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:06:14.712 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-19T22:06:14.712 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-19T22:06:14.713 INFO:teuthology.orchestra.run.smithi184.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-19T22:06:14.713 INFO:teuthology.orchestra.run.smithi184.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-19T22:06:14.713 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-19T22:06:14.713 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-19T22:06:14.713 INFO:teuthology.orchestra.run.smithi184.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-19T22:06:14.713 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-19T22:06:14.713 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-19T22:06:14.713 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-19T22:06:14.713 INFO:teuthology.orchestra.run.smithi184.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-19T22:06:14.713 INFO:teuthology.orchestra.run.smithi184.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-19T22:06:14.713 INFO:teuthology.orchestra.run.smithi184.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-19T22:06:14.714 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:14.714 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:14.834 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T22:06:14.947 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-19T22:06:15.280 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-19T22:06:15.529 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-19T22:06:15.588 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:01:57 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:06:15.733 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:06:16.024 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:06:16.025 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:06:16.025 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-19T22:06:16.025 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-19T22:06:16.025 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-19T22:06:16.025 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-19T22:06:16.454 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-19T22:06:16.454 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:16.454 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:06:16.454 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-19T22:06:16.455 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-19T22:06:16.455 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-19T22:06:16.455 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:16.455 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:16.604 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-fuse 2024-02-19T22:06:17.128 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:06:17.129 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:17.129 INFO:teuthology.orchestra.run.smithi184.stdout: Package Arch Version Repository Size 2024-02-19T22:06:17.129 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:17.129 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:06:17.129 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-19T22:06:17.129 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-02-19T22:06:17.129 INFO:teuthology.orchestra.run.smithi184.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-19T22:06:17.129 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-19T22:06:17.129 INFO:teuthology.orchestra.run.smithi184.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-19T22:06:17.129 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-19T22:06:17.129 INFO:teuthology.orchestra.run.smithi184.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-19T22:06:17.129 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:17.130 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:06:17.130 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:17.130 INFO:teuthology.orchestra.run.smithi184.stdout:Install 6 Packages 2024-02-19T22:06:17.130 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:17.130 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 5.9 M 2024-02-19T22:06:17.130 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 24 M 2024-02-19T22:06:17.130 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:06:17.362 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:04 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:06:17.427 INFO:teuthology.orchestra.run.smithi184.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 89 kB 00:00 2024-02-19T22:06:17.544 INFO:teuthology.orchestra.run.smithi184.stdout:(2/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 3.4 MB/s | 578 kB 00:00 2024-02-19T22:06:17.652 INFO:teuthology.orchestra.run.smithi184.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 94 kB/s | 21 kB 00:00 2024-02-19T22:06:17.761 INFO:teuthology.orchestra.run.smithi184.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 452 kB/s | 49 kB 00:00 2024-02-19T22:06:17.903 INFO:teuthology.orchestra.run.smithi184.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 992 kB/s | 355 kB 00:00 2024-02-19T22:06:18.261 INFO:teuthology.orchestra.run.smithi184.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.4 MB/s | 4.8 MB 00:00 2024-02-19T22:06:18.263 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:18.263 INFO:teuthology.orchestra.run.smithi184.stdout:Total 5.2 MB/s | 5.9 MB 00:01 2024-02-19T22:06:18.316 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:06:18.325 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:06:18.325 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:06:18.426 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:06:18.429 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:06:18.744 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:06:18.924 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:18.925 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:18.925 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-02-19T22:06:18.925 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:18.925 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:06:18.925 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-19T22:06:18.925 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-02-19T22:06:18.925 INFO:teuthology.orchestra.run.smithi176.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-19T22:06:18.925 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:18.925 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:06:18.926 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:18.926 INFO:teuthology.orchestra.run.smithi176.stdout:Install 2 Packages 2024-02-19T22:06:18.926 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:18.926 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 968 k 2024-02-19T22:06:18.926 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 3.2 M 2024-02-19T22:06:18.926 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:06:18.995 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-19T22:06:19.184 INFO:teuthology.orchestra.run.smithi176.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 14 MB/s | 885 kB 00:00 2024-02-19T22:06:19.325 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-19T22:06:19.351 INFO:teuthology.orchestra.run.smithi176.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 361 kB/s | 83 kB 00:00 2024-02-19T22:06:19.351 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:19.352 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.2 MB/s | 968 kB 00:00 2024-02-19T22:06:19.362 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:06:19.383 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:06:19.383 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:06:19.471 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:06:19.474 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:06:19.665 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:06:19.916 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-19T22:06:20.116 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:06:20.243 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-19T22:06:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:06:20.522 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-19T22:06:20.590 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-19T22:06:20.817 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-19T22:06:20.946 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-19T22:06:20.946 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:20.946 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:06:20.946 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-19T22:06:20.946 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:20.946 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:21.020 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:06:21.166 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados-devel 2024-02-19T22:06:21.327 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:06:21.327 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:06:21.327 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-19T22:06:21.327 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-19T22:06:21.327 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-19T22:06:21.327 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-19T22:06:21.935 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:08 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:06:22.016 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-19T22:06:22.016 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:22.016 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:06:22.017 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-19T22:06:22.017 INFO:teuthology.orchestra.run.smithi184.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-19T22:06:22.017 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-19T22:06:22.017 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:22.017 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:22.169 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install ceph-fuse 2024-02-19T22:06:22.932 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:04 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:06:23.496 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:23.497 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:23.497 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-02-19T22:06:23.497 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:23.497 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:06:23.497 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-19T22:06:23.497 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:23.497 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:06:23.497 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:23.497 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-02-19T22:06:23.497 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:23.498 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 131 k 2024-02-19T22:06:23.498 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 439 k 2024-02-19T22:06:23.498 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:06:23.538 INFO:teuthology.orchestra.run.smithi176.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.2 MB/s | 131 kB 00:00 2024-02-19T22:06:23.538 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:23.538 INFO:teuthology.orchestra.run.smithi176.stdout:Total 3.1 MB/s | 131 kB 00:00 2024-02-19T22:06:23.539 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:06:23.544 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:06:23.544 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:06:23.593 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:06:23.596 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:06:23.716 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:06:23.964 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:24.268 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:24.485 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:06:24.485 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-02-19T22:06:24.486 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:24.486 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:06:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-19T22:06:24.486 INFO:teuthology.orchestra.run.smithi184.stdout:Installing dependencies: 2024-02-19T22:06:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-19T22:06:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:24.486 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:06:24.486 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:24.486 INFO:teuthology.orchestra.run.smithi184.stdout:Install 2 Packages 2024-02-19T22:06:24.486 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:24.487 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 968 k 2024-02-19T22:06:24.487 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 3.2 M 2024-02-19T22:06:24.487 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:06:24.673 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:24.673 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:24.673 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:06:24.673 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:24.673 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:24.673 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:24.739 INFO:teuthology.orchestra.run.smithi184.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 13 MB/s | 885 kB 00:00 2024-02-19T22:06:24.906 INFO:teuthology.orchestra.run.smithi184.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 352 kB/s | 83 kB 00:00 2024-02-19T22:06:24.906 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:24.907 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.2 MB/s | 968 kB 00:00 2024-02-19T22:06:24.917 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:06:24.919 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs2 2024-02-19T22:06:24.937 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:06:24.938 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:06:25.025 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:06:25.028 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:06:25.198 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:06:25.488 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-19T22:06:25.669 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:06:25.699 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:12 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:06:26.068 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:06:26.068 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-19T22:06:26.475 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-19T22:06:26.475 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:26.475 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:06:26.475 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-19T22:06:26.475 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:26.475 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:26.714 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados-devel 2024-02-19T22:06:27.198 INFO:teuthology.orchestra.run.smithi176.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:27.245 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:27.246 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-02-19T22:06:27.246 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:27.367 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs-devel 2024-02-19T22:06:27.483 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:09 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:06:28.131 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:15 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:06:29.047 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:06:29.047 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:29.048 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-02-19T22:06:29.048 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:29.048 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:06:29.048 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-19T22:06:29.048 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:29.048 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:06:29.048 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:29.048 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-02-19T22:06:29.048 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:29.049 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 131 k 2024-02-19T22:06:29.049 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 439 k 2024-02-19T22:06:29.049 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:06:29.101 INFO:teuthology.orchestra.run.smithi184.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.5 MB/s | 131 kB 00:00 2024-02-19T22:06:29.101 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:29.101 INFO:teuthology.orchestra.run.smithi184.stdout:Total 2.4 MB/s | 131 kB 00:00 2024-02-19T22:06:29.102 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:06:29.107 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:06:29.107 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:06:29.156 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:06:29.159 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:06:29.265 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:06:29.476 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:29.690 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:29.690 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:29.690 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-02-19T22:06:29.691 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:29.691 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:06:29.691 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-19T22:06:29.691 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:29.691 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:06:29.691 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:29.691 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-02-19T22:06:29.691 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:29.692 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 24 k 2024-02-19T22:06:29.692 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 94 k 2024-02-19T22:06:29.692 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:06:29.744 INFO:teuthology.orchestra.run.smithi176.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 469 kB/s | 24 kB 00:00 2024-02-19T22:06:29.744 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:29.744 INFO:teuthology.orchestra.run.smithi176.stdout:Total 457 kB/s | 24 kB 00:00 2024-02-19T22:06:29.745 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:06:29.748 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:06:29.748 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:06:29.750 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:06:29.753 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:06:29.768 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:29.807 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:06:30.010 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:30.125 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:30.126 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:30.126 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:06:30.126 INFO:teuthology.orchestra.run.smithi184.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:30.126 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:30.126 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:30.280 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:30.330 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs2 2024-02-19T22:06:30.644 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:30.644 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:30.644 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:06:30.644 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:30.645 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:30.645 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:30.777 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados2 2024-02-19T22:06:31.096 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:13 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:06:31.539 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:18 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:06:32.589 INFO:teuthology.orchestra.run.smithi184.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:32.635 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:06:32.636 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-02-19T22:06:32.636 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:33.035 INFO:teuthology.orchestra.run.smithi176.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:33.083 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:33.084 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-02-19T22:06:33.084 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:33.205 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librbd1 2024-02-19T22:06:33.239 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install libcephfs-devel 2024-02-19T22:06:33.963 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:20 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:06:34.004 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:16 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:06:35.461 INFO:teuthology.orchestra.run.smithi176.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:35.507 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:35.508 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-02-19T22:06:35.508 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:35.564 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:06:35.564 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:35.565 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-02-19T22:06:35.565 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:35.565 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:06:35.565 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-19T22:06:35.565 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:35.565 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:06:35.565 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:35.565 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-02-19T22:06:35.565 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:35.566 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 24 k 2024-02-19T22:06:35.566 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 94 k 2024-02-19T22:06:35.566 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:06:35.625 INFO:teuthology.orchestra.run.smithi184.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 410 kB/s | 24 kB 00:00 2024-02-19T22:06:35.625 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:35.626 INFO:teuthology.orchestra.run.smithi184.stdout:Total 401 kB/s | 24 kB 00:00 2024-02-19T22:06:35.626 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:06:35.629 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:06:35.629 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:06:35.632 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:06:35.633 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rados 2024-02-19T22:06:35.635 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:06:35.683 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:06:35.900 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:36.165 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:36.392 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:23 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:06:36.513 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:36.513 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:36.513 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:06:36.513 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:36.513 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:36.513 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:36.673 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librados2 2024-02-19T22:06:37.430 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:19 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:06:37.909 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:37.955 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:37.956 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-02-19T22:06:37.957 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:38.090 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rgw 2024-02-19T22:06:38.844 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:25 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:06:38.948 INFO:teuthology.orchestra.run.smithi184.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:38.996 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:06:38.996 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-02-19T22:06:38.996 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:39.110 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install librbd1 2024-02-19T22:06:39.859 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:21 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:06:40.356 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:40.403 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:40.404 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-02-19T22:06:40.405 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:40.531 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-cephfs 2024-02-19T22:06:41.297 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:28 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:06:41.386 INFO:teuthology.orchestra.run.smithi184.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:41.434 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:06:41.435 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-02-19T22:06:41.435 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:41.598 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rados 2024-02-19T22:06:42.356 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:24 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:06:42.790 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:42.836 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:42.837 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-02-19T22:06:42.837 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:42.958 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rbd 2024-02-19T22:06:43.714 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:30 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:06:43.852 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:43.899 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:06:43.899 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-02-19T22:06:43.900 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:44.012 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rgw 2024-02-19T22:06:44.767 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:26 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:06:45.199 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:45.246 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:45.247 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-02-19T22:06:45.247 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:45.355 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-fuse 2024-02-19T22:06:46.115 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:33 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:06:46.272 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:46.318 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:06:46.319 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-02-19T22:06:46.319 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:46.429 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-cephfs 2024-02-19T22:06:47.178 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:29 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:06:47.676 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:47.677 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:47.677 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-02-19T22:06:47.677 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:47.677 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:06:47.677 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-19T22:06:47.677 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:47.677 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:06:47.677 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:47.677 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-02-19T22:06:47.678 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:47.678 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 89 k 2024-02-19T22:06:47.678 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 298 k 2024-02-19T22:06:47.678 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:06:47.724 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 89 kB 00:00 2024-02-19T22:06:47.725 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:47.725 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.8 MB/s | 89 kB 00:00 2024-02-19T22:06:47.725 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:06:47.734 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:06:47.734 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:06:47.783 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:06:47.786 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:06:47.891 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:06:48.052 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:48.352 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:48.660 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:48.707 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:06:48.708 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-02-19T22:06:48.708 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:48.730 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:48.730 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:48.731 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:06:48.731 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:48.731 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:48.731 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:48.821 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install python3-rbd 2024-02-19T22:06:48.985 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-mirror 2024-02-19T22:06:49.577 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:31 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:06:49.762 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:36 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:06:51.079 INFO:teuthology.orchestra.run.smithi184.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:06:51.125 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:06:51.126 INFO:teuthology.orchestra.run.smithi184.stdout:Nothing to do. 2024-02-19T22:06:51.126 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:51.246 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-fuse 2024-02-19T22:06:51.328 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:51.329 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:51.329 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-02-19T22:06:51.329 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:51.329 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:06:51.329 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:06:51.329 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:51.329 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:06:51.330 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:51.330 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-02-19T22:06:51.330 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:51.330 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 3.7 M 2024-02-19T22:06:51.330 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 18 M 2024-02-19T22:06:51.330 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:06:51.442 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 33 MB/s | 3.7 MB 00:00 2024-02-19T22:06:51.443 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:51.443 INFO:teuthology.orchestra.run.smithi176.stdout:Total 33 MB/s | 3.7 MB 00:00 2024-02-19T22:06:51.444 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:06:51.455 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:06:51.455 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:06:51.520 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:06:51.523 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:06:51.992 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:06:52.006 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:34 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:06:52.211 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:52.999 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:52.999 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:52.999 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:06:52.999 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:52.999 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:52.999 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:53.201 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-nbd 2024-02-19T22:06:53.560 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:06:53.561 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:53.561 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-02-19T22:06:53.561 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:53.561 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:06:53.561 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-19T22:06:53.561 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:53.562 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:06:53.562 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:53.562 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-02-19T22:06:53.562 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:53.562 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 89 k 2024-02-19T22:06:53.562 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 298 k 2024-02-19T22:06:53.563 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:06:53.614 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.7 MB/s | 89 kB 00:00 2024-02-19T22:06:53.614 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:53.614 INFO:teuthology.orchestra.run.smithi184.stdout:Total 1.7 MB/s | 89 kB 00:00 2024-02-19T22:06:53.615 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:06:53.624 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:06:53.624 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:06:53.673 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:06:53.676 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:06:53.783 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:06:53.972 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:40 ago on Mon 19 Feb 2024 10:04:13 PM UTC. 2024-02-19T22:06:53.997 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:54.285 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:54.662 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:54.662 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:54.662 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:06:54.662 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:54.662 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:54.662 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:54.901 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-mirror 2024-02-19T22:06:55.532 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-19T22:06:55.532 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:55.533 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-02-19T22:06:55.533 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:55.533 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-19T22:06:55.533 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-19T22:06:55.533 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:55.533 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-19T22:06:55.533 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-19T22:06:55.533 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-02-19T22:06:55.533 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:55.534 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 173 k 2024-02-19T22:06:55.534 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 649 k 2024-02-19T22:06:55.534 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-19T22:06:55.573 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.4 MB/s | 173 kB 00:00 2024-02-19T22:06:55.573 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:55.573 INFO:teuthology.orchestra.run.smithi176.stdout:Total 4.3 MB/s | 173 kB 00:00 2024-02-19T22:06:55.574 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-19T22:06:55.584 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-19T22:06:55.584 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-19T22:06:55.635 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-19T22:06:55.638 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-19T22:06:55.666 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:37 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:06:55.750 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-19T22:06:55.998 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:56.293 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:56.714 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:56.714 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:56.714 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-19T22:06:56.714 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:56.714 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-19T22:06:56.714 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-19T22:06:56.921 DEBUG:teuthology.parallel:result is None 2024-02-19T22:06:57.218 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:06:57.219 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:57.219 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-02-19T22:06:57.219 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:57.219 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:06:57.219 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:06:57.219 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:57.219 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:06:57.219 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:06:57.219 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-02-19T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 3.7 M 2024-02-19T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 18 M 2024-02-19T22:06:57.220 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:06:57.334 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 33 MB/s | 3.7 MB 00:00 2024-02-19T22:06:57.334 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:06:57.335 INFO:teuthology.orchestra.run.smithi184.stdout:Total 32 MB/s | 3.7 MB 00:00 2024-02-19T22:06:57.335 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:06:57.346 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:06:57.346 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:06:57.410 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:06:57.413 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:06:57.875 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:06:58.081 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:58.478 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:58.864 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:06:58.865 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:58.865 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:06:58.865 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-19T22:06:58.865 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:06:58.865 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:06:59.157 DEBUG:teuthology.orchestra.run.smithi184:> sudo yum -y install rbd-nbd 2024-02-19T22:06:59.928 INFO:teuthology.orchestra.run.smithi184.stdout:Last metadata expiration check: 0:02:41 ago on Mon 19 Feb 2024 10:04:18 PM UTC. 2024-02-19T22:07:01.466 INFO:teuthology.orchestra.run.smithi184.stdout:Dependencies resolved. 2024-02-19T22:07:01.466 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:07:01.466 INFO:teuthology.orchestra.run.smithi184.stdout: Package Architecture Version Repository Size 2024-02-19T22:07:01.467 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:07:01.467 INFO:teuthology.orchestra.run.smithi184.stdout:Installing: 2024-02-19T22:07:01.467 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-19T22:07:01.467 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:07:01.467 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction Summary 2024-02-19T22:07:01.467 INFO:teuthology.orchestra.run.smithi184.stdout:================================================================================ 2024-02-19T22:07:01.467 INFO:teuthology.orchestra.run.smithi184.stdout:Install 1 Package 2024-02-19T22:07:01.467 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:07:01.468 INFO:teuthology.orchestra.run.smithi184.stdout:Total download size: 173 k 2024-02-19T22:07:01.468 INFO:teuthology.orchestra.run.smithi184.stdout:Installed size: 649 k 2024-02-19T22:07:01.468 INFO:teuthology.orchestra.run.smithi184.stdout:Downloading Packages: 2024-02-19T22:07:01.510 INFO:teuthology.orchestra.run.smithi184.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.0 MB/s | 173 kB 00:00 2024-02-19T22:07:01.511 INFO:teuthology.orchestra.run.smithi184.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:07:01.511 INFO:teuthology.orchestra.run.smithi184.stdout:Total 3.9 MB/s | 173 kB 00:00 2024-02-19T22:07:01.511 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction check 2024-02-19T22:07:01.521 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction check succeeded. 2024-02-19T22:07:01.521 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction test 2024-02-19T22:07:01.570 INFO:teuthology.orchestra.run.smithi184.stdout:Transaction test succeeded. 2024-02-19T22:07:01.574 INFO:teuthology.orchestra.run.smithi184.stdout:Running transaction 2024-02-19T22:07:01.698 INFO:teuthology.orchestra.run.smithi184.stdout: Preparing : 1/1 2024-02-19T22:07:01.944 INFO:teuthology.orchestra.run.smithi184.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:02.245 INFO:teuthology.orchestra.run.smithi184.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:02.643 INFO:teuthology.orchestra.run.smithi184.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:07:02.643 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:07:02.643 INFO:teuthology.orchestra.run.smithi184.stdout:Installed: 2024-02-19T22:07:02.643 INFO:teuthology.orchestra.run.smithi184.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:07:02.644 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:07:02.644 INFO:teuthology.orchestra.run.smithi184.stdout:Complete! 2024-02-19T22:07:02.891 DEBUG:teuthology.parallel:result is None 2024-02-19T22:07:02.892 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:07:02.892 INFO:teuthology.packaging:ref: None 2024-02-19T22:07:02.892 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:07:02.892 INFO:teuthology.packaging:branch: None 2024-02-19T22:07:02.892 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:07:02.892 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:07:03.042 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T22:07:03.068 INFO:teuthology.orchestra.run.smithi176.stdout:16.2.4-0.el8 2024-02-19T22:07:03.069 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-19T22:07:03.069 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-19T22:07:03.071 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:07:03.071 INFO:teuthology.packaging:ref: None 2024-02-19T22:07:03.071 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:07:03.071 INFO:teuthology.packaging:branch: None 2024-02-19T22:07:03.071 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:07:03.071 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:07:03.213 DEBUG:teuthology.orchestra.run.smithi184:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T22:07:03.236 INFO:teuthology.orchestra.run.smithi184.stdout:16.2.4-0.el8 2024-02-19T22:07:03.236 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-19T22:07:03.236 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-19T22:07:03.238 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-19T22:07:03.238 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-19T22:07:03.238 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T22:07:03.272 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-02-19T22:07:03.272 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T22:07:03.303 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-19T22:07:03.304 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-19T22:07:03.304 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T22:07:03.344 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T22:07:03.415 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-02-19T22:07:03.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T22:07:03.442 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T22:07:03.509 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-19T22:07:03.509 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-19T22:07:03.509 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T22:07:03.541 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T22:07:03.611 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-02-19T22:07:03.611 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T22:07:03.638 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T22:07:03.704 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-19T22:07:03.704 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-19T22:07:03.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T22:07:03.735 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T22:07:03.805 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-02-19T22:07:03.805 DEBUG:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T22:07:03.831 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T22:07:03.897 INFO:teuthology.run_tasks:Running task print... 2024-02-19T22:07:03.912 INFO:teuthology.task.print:**** done install task... 2024-02-19T22:07:03.912 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-19T22:07:04.028 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(PG_DEGRADED\\)', 'Degraded data redundancy', '\\(PG_', 'MDS_INSUFFICIENT_STANDBY', 'deprecated feature inline_data', 'compat changed unexpectedly', 'overall HEALTH_', 'OSD_DOWN', 'OSD_', 'but it is still running', 'is not responding'], 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'} 2024-02-19T22:07:04.028 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-02-19T22:07:04.029 INFO:tasks.cephadm:Cluster fsid is 36a8f1ec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:07:04.029 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-19T22:07:04.029 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-19T22:07:04.029 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi176': '172.21.15.176', 'mon.smithi184': '172.21.15.184'} 2024-02-19T22:07:04.030 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-19T22:07:04.030 DEBUG:teuthology.orchestra.run.smithi176:> sudo hostname $(hostname -s) 2024-02-19T22:07:04.061 DEBUG:teuthology.orchestra.run.smithi184:> sudo hostname $(hostname -s) 2024-02-19T22:07:04.088 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-02-19T22:07:04.088 DEBUG:teuthology.orchestra.run.smithi176:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T22:07:04.311 INFO:teuthology.orchestra.run.smithi176.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 19 22:07 /home/ubuntu/cephtest/cephadm 2024-02-19T22:07:04.311 DEBUG:teuthology.orchestra.run.smithi184:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T22:07:04.453 INFO:teuthology.orchestra.run.smithi184.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 19 22:07 /home/ubuntu/cephtest/cephadm 2024-02-19T22:07:04.454 DEBUG:teuthology.orchestra.run.smithi176:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T22:07:04.476 DEBUG:teuthology.orchestra.run.smithi184:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T22:07:04.504 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph 2024-02-19T22:07:04.546 DEBUG:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph 2024-02-19T22:07:04.572 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /etc/ceph 2024-02-19T22:07:04.615 DEBUG:teuthology.orchestra.run.smithi184:> sudo chmod 777 /etc/ceph 2024-02-19T22:07:04.640 INFO:tasks.cephadm:Writing seed config... 2024-02-19T22:07:04.641 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-19T22:07:04.641 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-19T22:07:04.641 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-19T22:07:04.641 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-19T22:07:04.642 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-19T22:07:04.642 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-19T22:07:04.642 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-19T22:07:04.642 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-19T22:07:04.642 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-19T22:07:04.642 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-19T22:07:04.642 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-19T22:07:04.642 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-19T22:07:04.642 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-19T22:07:04.642 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-19T22:07:04.642 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-19T22:07:04.642 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-19T22:07:04.642 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-19T22:07:04.642 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-19T22:07:04.643 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-19T22:07:04.644 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-19T22:07:04.644 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-19T22:07:04.644 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-19T22:07:04.644 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-19T22:07:04.644 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-19T22:07:04.644 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-19T22:07:04.645 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-19T22:07:04.674 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 36a8f1ec-cf73-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-19T22:07:04.675 DEBUG:teuthology.orchestra.run.smithi176:mon.smithi176> sudo journalctl -f -n 0 -u ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi176.service 2024-02-19T22:07:04.716 INFO:tasks.cephadm:Bootstrapping... 2024-02-19T22:07:04.717 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.176 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-19T22:07:04.745 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:-- Logs begin at Mon 2024-02-19 21:56:55 UTC. -- 2024-02-19T22:07:05.021 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T22:07:05.981 INFO:teuthology.orchestra.run.smithi176.stderr:/bin/podman: 4.9.3-dev 2024-02-19T22:07:05.995 INFO:teuthology.orchestra.run.smithi176.stderr:Verifying podman|docker is present... 2024-02-19T22:07:05.995 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T22:07:06.019 INFO:teuthology.orchestra.run.smithi176.stderr:/bin/podman: 4.9.3-dev 2024-02-19T22:07:06.032 INFO:teuthology.orchestra.run.smithi176.stderr:Verifying lvm2 is present... 2024-02-19T22:07:06.033 INFO:teuthology.orchestra.run.smithi176.stderr:Verifying time synchronization is in place... 2024-02-19T22:07:06.033 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-enabled chrony.service 2024-02-19T22:07:06.040 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T22:07:06.041 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-active chrony.service 2024-02-19T22:07:06.047 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: inactive 2024-02-19T22:07:06.090 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-19T22:07:06.090 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: enabled 2024-02-19T22:07:06.090 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-active chronyd.service 2024-02-19T22:07:06.090 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: active 2024-02-19T22:07:06.090 INFO:teuthology.orchestra.run.smithi176.stderr:Unit chronyd.service is enabled and running 2024-02-19T22:07:06.090 INFO:teuthology.orchestra.run.smithi176.stderr:Repeating the final host check... 2024-02-19T22:07:06.091 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T22:07:06.091 INFO:teuthology.orchestra.run.smithi176.stderr:/bin/podman: 4.9.3-dev 2024-02-19T22:07:06.092 INFO:teuthology.orchestra.run.smithi176.stderr:podman|docker (/bin/podman) is present 2024-02-19T22:07:06.092 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl is present 2024-02-19T22:07:06.092 INFO:teuthology.orchestra.run.smithi176.stderr:lvcreate is present 2024-02-19T22:07:06.092 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-enabled chrony.service 2024-02-19T22:07:06.098 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T22:07:06.099 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-active chrony.service 2024-02-19T22:07:06.104 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: inactive 2024-02-19T22:07:06.104 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-19T22:07:06.110 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: enabled 2024-02-19T22:07:06.111 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-active chronyd.service 2024-02-19T22:07:06.116 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: active 2024-02-19T22:07:06.116 INFO:teuthology.orchestra.run.smithi176.stderr:Unit chronyd.service is enabled and running 2024-02-19T22:07:06.116 INFO:teuthology.orchestra.run.smithi176.stderr:Host looks OK 2024-02-19T22:07:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:Cluster fsid: 36a8f1ec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:07:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:Acquiring lock 139877961636328 on /run/cephadm/36a8f1ec-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:07:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:Lock 139877961636328 acquired on /run/cephadm/36a8f1ec-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:07:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:Verifying IP 172.21.15.176 port 3300 ... 2024-02-19T22:07:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:Verifying IP 172.21.15.176 port 6789 ... 2024-02-19T22:07:06.118 INFO:teuthology.orchestra.run.smithi176.stderr:Base mon IP is 172.21.15.176, final addrv is [v2:172.21.15.176:3300,v1:172.21.15.176:6789] 2024-02-19T22:07:06.118 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /sbin/ip route ls 2024-02-19T22:07:06.120 INFO:teuthology.orchestra.run.smithi176.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.176 metric 100 2024-02-19T22:07:06.120 INFO:teuthology.orchestra.run.smithi176.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.176 metric 100 2024-02-19T22:07:06.121 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /sbin/ip -6 route ls 2024-02-19T22:07:06.123 INFO:teuthology.orchestra.run.smithi176.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-19T22:07:06.123 INFO:teuthology.orchestra.run.smithi176.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-19T22:07:06.123 INFO:teuthology.orchestra.run.smithi176.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1796sec hoplimit 64 pref medium 2024-02-19T22:07:06.123 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /sbin/ip -6 addr ls 2024-02-19T22:07:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-19T22:07:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-19T22:07:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-19T22:07:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-19T22:07:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:6f61/64 scope link 2024-02-19T22:07:06.126 INFO:teuthology.orchestra.run.smithi176.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-19T22:07:06.127 INFO:teuthology.orchestra.run.smithi176.stderr:Mon IP 172.21.15.176 is in CIDR network 172.21.0.0/20 2024-02-19T22:07:06.127 INFO:teuthology.orchestra.run.smithi176.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-19T22:07:06.128 INFO:teuthology.orchestra.run.smithi176.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-02-19T22:07:06.128 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-02-19T22:07:06.221 INFO:teuthology.orchestra.run.smithi176.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-02-19T22:07:12.965 INFO:teuthology.orchestra.run.smithi176.stderr:/bin/podman: Getting image source signatures 2024-02-19T22:07:12.974 INFO:teuthology.orchestra.run.smithi176.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-02-19T22:07:12.975 INFO:teuthology.orchestra.run.smithi176.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-19T22:07:25.738 INFO:teuthology.orchestra.run.smithi176.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-19T22:07:25.897 INFO:teuthology.orchestra.run.smithi176.stderr:/bin/podman: Writing manifest to image destination 2024-02-19T22:07:26.124 INFO:teuthology.orchestra.run.smithi176.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-19T22:07:26.137 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-02-19T22:07:26.769 INFO:teuthology.orchestra.run.smithi176.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-19T22:07:27.425 INFO:teuthology.orchestra.run.smithi176.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-19T22:07:27.425 INFO:teuthology.orchestra.run.smithi176.stderr:Extracting ceph user uid/gid from container image... 2024-02-19T22:07:27.425 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-02-19T22:07:27.909 INFO:teuthology.orchestra.run.smithi176.stderr:stat: 167 167 2024-02-19T22:07:28.402 INFO:teuthology.orchestra.run.smithi176.stderr:Creating initial keys... 2024-02-19T22:07:28.403 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T22:07:28.906 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-authtool: AQAg0dNlstSAMBAAvQES7m8NVG0H2/wb1RjUDQ== 2024-02-19T22:07:29.443 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T22:07:29.958 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-authtool: AQAh0dNlkgUMMhAAfBQ1vDtSnosCAonzJz+yQg== 2024-02-19T22:07:30.524 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T22:07:31.017 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-authtool: AQAi0dNl6tVhNxAAX2YkwMvdF5NHcEjpBOJdUA== 2024-02-19T22:07:31.504 INFO:teuthology.orchestra.run.smithi176.stderr:Creating initial monmap... 2024-02-19T22:07:31.504 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpktx9z7ve:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 --addv smithi176 [v2:172.21.15.176:3300,v1:172.21.15.176:6789] /tmp/monmap 2024-02-19T22:07:32.091 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T22:07:32.091 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 36a8f1ec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:07:32.091 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T22:07:32.659 INFO:teuthology.orchestra.run.smithi176.stderr:monmaptool for smithi176 [v2:172.21.15.176:3300,v1:172.21.15.176:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T22:07:32.660 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/monmaptool: set fsid to 36a8f1ec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:07:32.660 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T22:07:32.660 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-02-19T22:07:32.660 INFO:teuthology.orchestra.run.smithi176.stderr:Creating mon... 2024-02-19T22:07:32.660 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/mon.smithi176:/var/lib/ceph/mon/ceph-smithi176:z -v /tmp/ceph-tmp8bbpd9kz:/tmp/keyring:z -v /tmp/ceph-tmpktx9z7ve:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi176 --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-19T22:07:33.272 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.138+0000 7f92fbe1e700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T22:07:33.272 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.138+0000 7f92fbe1e700 1 imported monmap: 2024-02-19T22:07:33.272 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-19T22:07:33.272 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:07:33.272 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: last_changed 2024-02-19T22:07:31.982550+0000 2024-02-19T22:07:33.272 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: created 2024-02-19T22:07:31.982550+0000 2024-02-19T22:07:33.272 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-19T22:07:33.273 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-19T22:07:33.273 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.smithi176 2024-02-19T22:07:33.273 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.273 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 0 /usr/bin/ceph-mon: set fsid to 36a8f1ec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:07:33.273 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T22:07:33.273 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.273 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T22:07:33.273 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Compile date May 13 2021 2024-02-19T22:07:33.273 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: DB SUMMARY 2024-02-19T22:07:33.273 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.273 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi176/store.db dir, Total Num: 0, files: 2024-02-19T22:07:33.273 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.273 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi176/store.db: 2024-02-19T22:07:33.273 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.274 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.error_if_exists: 0 2024-02-19T22:07:33.274 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.create_if_missing: 1 2024-02-19T22:07:33.274 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T22:07:33.274 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.env: 0x55d2711611c0 2024-02-19T22:07:33.274 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.fs: Posix File System 2024-02-19T22:07:33.274 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.info_log: 0x55d27364cec0 2024-02-19T22:07:33.274 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T22:07:33.274 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.statistics: (nil) 2024-02-19T22:07:33.274 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.use_fsync: 0 2024-02-19T22:07:33.274 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T22:07:33.274 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T22:07:33.274 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T22:07:33.274 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T22:07:33.274 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T22:07:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T22:07:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T22:07:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T22:07:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T22:07:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T22:07:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T22:07:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.db_log_dir: 2024-02-19T22:07:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi176/store.db 2024-02-19T22:07:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T22:07:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T22:07:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T22:07:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T22:07:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T22:07:33.276 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T22:07:33.276 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T22:07:33.276 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T22:07:33.276 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T22:07:33.276 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T22:07:33.276 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.write_buffer_manager: 0x55d273653530 2024-02-19T22:07:33.276 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T22:07:33.276 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T22:07:33.276 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T22:07:33.276 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T22:07:33.276 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T22:07:33.277 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T22:07:33.277 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T22:07:33.277 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T22:07:33.277 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T22:07:33.277 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.unordered_write: 0 2024-02-19T22:07:33.277 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T22:07:33.277 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T22:07:33.277 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T22:07:33.277 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T22:07:33.277 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.row_cache: None 2024-02-19T22:07:33.277 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.wal_filter: None 2024-02-19T22:07:33.277 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T22:07:33.278 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T22:07:33.278 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T22:07:33.278 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.two_write_queues: 0 2024-02-19T22:07:33.278 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T22:07:33.278 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.atomic_flush: 0 2024-02-19T22:07:33.278 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T22:07:33.278 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T22:07:33.278 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T22:07:33.278 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T22:07:33.278 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T22:07:33.278 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T22:07:33.278 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T22:07:33.278 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T22:07:33.278 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T22:07:33.279 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T22:07:33.279 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T22:07:33.279 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T22:07:33.279 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T22:07:33.279 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T22:07:33.279 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T22:07:33.279 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.max_open_files: -1 2024-02-19T22:07:33.279 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T22:07:33.279 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T22:07:33.279 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T22:07:33.279 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T22:07:33.279 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Compression algorithms supported: 2024-02-19T22:07:33.279 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T22:07:33.279 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: kZSTD supported: 0 2024-02-19T22:07:33.280 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: kXpressCompression supported: 0 2024-02-19T22:07:33.280 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T22:07:33.280 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T22:07:33.280 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T22:07:33.280 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: kZlibCompression supported: 1 2024-02-19T22:07:33.280 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T22:07:33.280 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.139+0000 7f92fbe1e700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T22:07:33.280 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-19T22:07:33.280 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.280 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi176/store.db/MANIFEST-000001 2024-02-19T22:07:33.280 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.280 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T22:07:33.280 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.281 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T22:07:33.281 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.merge_operator: 2024-02-19T22:07:33.281 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compaction_filter: None 2024-02-19T22:07:33.281 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T22:07:33.281 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T22:07:33.281 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T22:07:33.281 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d273565cb8) 2024-02-19T22:07:33.281 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-19T22:07:33.281 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T22:07:33.281 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T22:07:33.281 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-19T22:07:33.281 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-19T22:07:33.281 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-19T22:07:33.281 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: block_cache: 0x55d27359ad10 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-19T22:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-19T22:07:33.283 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-19T22:07:33.283 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-19T22:07:33.283 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-19T22:07:33.283 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-19T22:07:33.283 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-19T22:07:33.283 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T22:07:33.283 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-19T22:07:33.283 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-19T22:07:33.283 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-19T22:07:33.283 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-19T22:07:33.283 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-19T22:07:33.283 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-19T22:07:33.283 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.284 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T22:07:33.284 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T22:07:33.284 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compression: NoCompression 2024-02-19T22:07:33.284 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T22:07:33.284 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T22:07:33.284 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T22:07:33.284 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.num_levels: 7 2024-02-19T22:07:33.284 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T22:07:33.284 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T22:07:33.284 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T22:07:33.284 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T22:07:33.284 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T22:07:33.284 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T22:07:33.285 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T22:07:33.285 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:07:33.285 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T22:07:33.285 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T22:07:33.285 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T22:07:33.285 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T22:07:33.285 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T22:07:33.285 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:07:33.285 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T22:07:33.285 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T22:07:33.285 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T22:07:33.285 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T22:07:33.285 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T22:07:33.285 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T22:07:33.286 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T22:07:33.286 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T22:07:33.286 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T22:07:33.286 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T22:07:33.286 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T22:07:33.286 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T22:07:33.286 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T22:07:33.286 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T22:07:33.286 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T22:07:33.286 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T22:07:33.286 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T22:07:33.286 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T22:07:33.286 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T22:07:33.286 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.table_properties_collectors: 2024-02-19T22:07:33.287 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T22:07:33.288 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T22:07:33.288 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T22:07:33.288 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T22:07:33.288 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T22:07:33.288 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.bloom_locality: 0 2024-02-19T22:07:33.288 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T22:07:33.288 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T22:07:33.288 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T22:07:33.288 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T22:07:33.288 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T22:07:33.288 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.ttl: 2592000 2024-02-19T22:07:33.288 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.140+0000 7f92fbe1e700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T22:07:33.288 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.141+0000 7f92fbe1e700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi176/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T22:07:33.288 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.141+0000 7f92fbe1e700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.141+0000 7f92fbe1e700 4 rocksdb: DB pointer 0x55d27365f800 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.141+0000 7f92e4cb7700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.141+0000 7f92e4cb7700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.289 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:07:33.290 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:07:33.290 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:07:33.290 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:07:33.290 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:07:33.290 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.290 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:07:33.290 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:07:33.290 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:07:33.290 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:07:33.290 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:07:33.291 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:07:33.291 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:07:33.291 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:07:33.291 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:07:33.291 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:07:33.291 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:07:33.291 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:07:33.291 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.291 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:07:33.291 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.291 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:07:33.291 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:07:33.291 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:07:33.292 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:07:33.292 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:07:33.292 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.292 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:07:33.292 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:07:33.292 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:07:33.292 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:07:33.292 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:07:33.292 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:07:33.292 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:07:33.292 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:07:33.292 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:07:33.292 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:07:33.292 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:07:33.293 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:07:33.293 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.293 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:07:33.293 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: 2024-02-19T22:07:33.293 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.143+0000 7f92fbe1e700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-19T22:07:33.293 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.143+0000 7f92fbe1e700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-19T22:07:33.293 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:07:33.143+0000 7f92fbe1e700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi176 for mon.smithi176 2024-02-19T22:07:33.860 INFO:teuthology.orchestra.run.smithi176.stderr:create mon.smithi176 on 2024-02-19T22:07:33.861 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:07:33.866 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl disable ceph.target 2024-02-19T22:07:33.873 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-19T22:07:33.962 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl enable ceph.target 2024-02-19T22:07:33.969 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-19T22:07:34.052 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl start ceph.target 2024-02-19T22:07:34.059 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl enable ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715.target 2024-02-19T22:07:34.065 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715.target. 2024-02-19T22:07:34.065 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715.target. 2024-02-19T22:07:34.144 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl start ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715.target 2024-02-19T22:07:34.152 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl daemon-reload 2024-02-19T22:07:34.235 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl stop ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi176 2024-02-19T22:07:34.258 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl reset-failed ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi176 2024-02-19T22:07:34.264 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: Failed to reset failed state of unit ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi176.service: Unit ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi176.service not loaded. 2024-02-19T22:07:34.264 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl enable ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi176 2024-02-19T22:07:34.270 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: Created symlink /etc/systemd/system/ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715.target.wants/ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi176.service → /etc/systemd/system/ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@.service. 2024-02-19T22:07:34.355 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl start ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi176 2024-02-19T22:07:34.825 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:07:34.833 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: disabled 2024-02-19T22:07:34.834 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:07:34.841 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: inactive 2024-02-19T22:07:34.842 INFO:teuthology.orchestra.run.smithi176.stderr:firewalld.service is not enabled 2024-02-19T22:07:34.842 INFO:teuthology.orchestra.run.smithi176.stderr:Not possible to enable service . firewalld.service is not available 2024-02-19T22:07:34.842 INFO:teuthology.orchestra.run.smithi176.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-19T22:07:34.842 INFO:teuthology.orchestra.run.smithi176.stderr:Waiting for mon to start... 2024-02-19T22:07:34.842 INFO:teuthology.orchestra.run.smithi176.stderr:Waiting for mon... 2024-02-19T22:07:34.842 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/mon.smithi176:/var/lib/ceph/mon/ceph-smithi176:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-02-19T22:07:35.098 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:34 smithi176 bash[37684]: 7114095eb59ab0dad7439f94c2a426b5b017ed7e8842f0225e67502f5096f0ab 2024-02-19T22:07:35.099 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:34 smithi176 systemd[1]: Started Ceph mon.smithi176 for 36a8f1ec-cf73-11ee-95bb-87774f69a715. 2024-02-19T22:07:35.380 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.377+0000 7f4a2207e700 1 Processor -- start 2024-02-19T22:07:35.380 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.377+0000 7f4a2207e700 1 -- start start 2024-02-19T22:07:35.380 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.377+0000 7f4a2207e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a1c0fad90 0x7f4a1c0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:35.380 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.377+0000 7f4a2207e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c0fb6a0 con 0x7f4a1c0fad90 2024-02-19T22:07:35.380 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.377+0000 7f4a1b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a1c0fad90 0x7f4a1c0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:35.380 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.377+0000 7f4a1b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a1c0fad90 0x7f4a1c0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:55212/0 (socket says 172.21.15.176:55212) 2024-02-19T22:07:35.380 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.377+0000 7f4a1b7fe700 1 -- 172.21.15.176:0/715317209 learned_addr learned my addr 172.21.15.176:0/715317209 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:35.380 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.378+0000 7f4a1b7fe700 1 -- 172.21.15.176:0/715317209 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c0fb7e0 con 0x7f4a1c0fad90 2024-02-19T22:07:35.380 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.378+0000 7f4a1b7fe700 1 --2- 172.21.15.176:0/715317209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a1c0fad90 0x7f4a1c0fb1b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f4a0c00b5f0 tx=0x7f4a0c006ee0).ready entity=mon.0 client_cookie=b3fd69c43462e61e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:35.381 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.378+0000 7f4a1a7fc700 1 -- 172.21.15.176:0/715317209 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a0c00d600 con 0x7f4a1c0fad90 2024-02-19T22:07:35.381 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.378+0000 7f4a1a7fc700 1 -- 172.21.15.176:0/715317209 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0c00d760 con 0x7f4a1c0fad90 2024-02-19T22:07:35.381 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.378+0000 7f4a1a7fc700 1 -- 172.21.15.176:0/715317209 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a0c00da30 con 0x7f4a1c0fad90 2024-02-19T22:07:35.381 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.378+0000 7f4a2207e700 1 -- 172.21.15.176:0/715317209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a1c0fad90 msgr2=0x7f4a1c0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:35.381 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.378+0000 7f4a2207e700 1 --2- 172.21.15.176:0/715317209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a1c0fad90 0x7f4a1c0fb1b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f4a0c00b5f0 tx=0x7f4a0c006ee0).stop 2024-02-19T22:07:35.381 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.378+0000 7f4a2207e700 1 -- 172.21.15.176:0/715317209 shutdown_connections 2024-02-19T22:07:35.381 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.378+0000 7f4a2207e700 1 --2- 172.21.15.176:0/715317209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a1c0fad90 0x7f4a1c0fb1b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:35.381 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.378+0000 7f4a2207e700 1 -- 172.21.15.176:0/715317209 >> 172.21.15.176:0/715317209 conn(0x7f4a1c0f6530 msgr2=0x7f4a1c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:35.381 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.378+0000 7f4a2207e700 1 -- 172.21.15.176:0/715317209 shutdown_connections 2024-02-19T22:07:35.381 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.379+0000 7f4a2207e700 1 -- 172.21.15.176:0/715317209 wait complete. 2024-02-19T22:07:35.381 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.379+0000 7f4a2207e700 1 Processor -- start 2024-02-19T22:07:35.381 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.379+0000 7f4a2207e700 1 -- start start 2024-02-19T22:07:35.383 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.379+0000 7f4a2207e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a1c0fad90 0x7f4a1c10d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:35.383 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.379+0000 7f4a2207e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c070450 con 0x7f4a1c0fad90 2024-02-19T22:07:35.383 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.380+0000 7f4a1b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a1c0fad90 0x7f4a1c10d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:35.383 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.380+0000 7f4a1b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a1c0fad90 0x7f4a1c10d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:55228/0 (socket says 172.21.15.176:55228) 2024-02-19T22:07:35.383 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.380+0000 7f4a1b7fe700 1 -- 172.21.15.176:0/1383978034 learned_addr learned my addr 172.21.15.176:0/1383978034 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:35.383 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.380+0000 7f4a1b7fe700 1 -- 172.21.15.176:0/1383978034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a0c00b200 con 0x7f4a1c0fad90 2024-02-19T22:07:35.384 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.380+0000 7f4a1b7fe700 1 --2- 172.21.15.176:0/1383978034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a1c0fad90 0x7f4a1c10d090 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4a0c006c70 tx=0x7f4a0c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:35.384 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.380+0000 7f4a03fff700 1 -- 172.21.15.176:0/1383978034 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a0c011b40 con 0x7f4a1c0fad90 2024-02-19T22:07:35.384 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.380+0000 7f4a03fff700 1 -- 172.21.15.176:0/1383978034 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0c011ca0 con 0x7f4a1c0fad90 2024-02-19T22:07:35.384 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.380+0000 7f4a2207e700 1 -- 172.21.15.176:0/1383978034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1c070650 con 0x7f4a1c0fad90 2024-02-19T22:07:35.384 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.380+0000 7f4a03fff700 1 -- 172.21.15.176:0/1383978034 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a0c01d430 con 0x7f4a1c0fad90 2024-02-19T22:07:35.384 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.380+0000 7f4a2207e700 1 -- 172.21.15.176:0/1383978034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1c10d740 con 0x7f4a1c0fad90 2024-02-19T22:07:35.384 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.381+0000 7f4a03fff700 1 -- 172.21.15.176:0/1383978034 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f4a0c00f070 con 0x7f4a1c0fad90 2024-02-19T22:07:35.384 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.381+0000 7f4a03fff700 1 -- 172.21.15.176:0/1383978034 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4a0c027460 con 0x7f4a1c0fad90 2024-02-19T22:07:35.384 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.381+0000 7f4a2207e700 1 -- 172.21.15.176:0/1383978034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a1c045ad0 con 0x7f4a1c0fad90 2024-02-19T22:07:35.384 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.384+0000 7f4a03fff700 1 -- 172.21.15.176:0/1383978034 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4a0c016070 con 0x7f4a1c0fad90 2024-02-19T22:07:35.454 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.451+0000 7f4a2207e700 1 -- 172.21.15.176:0/1383978034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f4a1c0fc050 con 0x7f4a1c0fad90 2024-02-19T22:07:35.454 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.451+0000 7f4a03fff700 1 -- 172.21.15.176:0/1383978034 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f4a0c01d640 con 0x7f4a1c0fad90 2024-02-19T22:07:35.454 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: cluster: 2024-02-19T22:07:35.454 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: id: 36a8f1ec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:07:35.454 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-19T22:07:35.455 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.455 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: services: 2024-02-19T22:07:35.455 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi176 (age 0.673474s) 2024-02-19T22:07:35.455 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-19T22:07:35.455 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-19T22:07:35.455 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.455 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: data: 2024-02-19T22:07:35.455 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-19T22:07:35.455 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-19T22:07:35.455 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-19T22:07:35.455 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: pgs: 2024-02-19T22:07:35.455 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.455 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.452+0000 7f4a2207e700 1 -- 172.21.15.176:0/1383978034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a1c0fad90 msgr2=0x7f4a1c10d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:35.455 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.452+0000 7f4a2207e700 1 --2- 172.21.15.176:0/1383978034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a1c0fad90 0x7f4a1c10d090 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4a0c006c70 tx=0x7f4a0c016000).stop 2024-02-19T22:07:35.456 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.452+0000 7f4a2207e700 1 -- 172.21.15.176:0/1383978034 shutdown_connections 2024-02-19T22:07:35.456 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.452+0000 7f4a2207e700 1 --2- 172.21.15.176:0/1383978034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4a1c0fad90 0x7f4a1c10d090 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:35.456 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.452+0000 7f4a2207e700 1 -- 172.21.15.176:0/1383978034 >> 172.21.15.176:0/1383978034 conn(0x7f4a1c0f6530 msgr2=0x7f4a1c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:35.456 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.452+0000 7f4a2207e700 1 -- 172.21.15.176:0/1383978034 shutdown_connections 2024-02-19T22:07:35.456 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:35.452+0000 7f4a2207e700 1 -- 172.21.15.176:0/1383978034 wait complete. 2024-02-19T22:07:35.970 INFO:teuthology.orchestra.run.smithi176.stderr:mon is available 2024-02-19T22:07:35.971 INFO:teuthology.orchestra.run.smithi176.stderr:Assimilating anything we can from ceph.conf... 2024-02-19T22:07:35.971 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z -v /var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/mon.smithi176:/var/lib/ceph/mon/ceph-smithi176:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi176/config 2024-02-19T22:07:36.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[37699]: cluster 2024-02-19T22:07:34.779803+0000 mon.smithi176 (mon.0) 2 : cluster [INF] mon.smithi176 is new leader, mons smithi176 in quorum (ranks 0) 2024-02-19T22:07:36.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[37699]: cluster 2024-02-19T22:07:34.779977+0000 mon.smithi176 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi176=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} 2024-02-19T22:07:36.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[37699]: cluster 2024-02-19T22:07:34.783097+0000 mon.smithi176 (mon.0) 4 : cluster [DBG] fsmap 2024-02-19T22:07:36.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[37699]: cluster 2024-02-19T22:07:34.786398+0000 mon.smithi176 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-19T22:07:36.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[37699]: cluster 2024-02-19T22:07:34.786834+0000 mon.smithi176 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-19T22:07:36.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[37699]: audit 2024-02-19T22:07:35.453247+0000 mon.smithi176 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.176:0/1383978034' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-19T22:07:36.467 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.464+0000 7f8c80f1b700 1 Processor -- start 2024-02-19T22:07:36.467 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.464+0000 7f8c80f1b700 1 -- start start 2024-02-19T22:07:36.467 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.464+0000 7f8c80f1b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c7c0703d0 0x7f8c7c0feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:36.467 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.464+0000 7f8c80f1b700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0fefa0 con 0x7f8c7c0703d0 2024-02-19T22:07:36.467 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.465+0000 7f8c7a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c7c0703d0 0x7f8c7c0feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:36.467 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.465+0000 7f8c7a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c7c0703d0 0x7f8c7c0feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:55236/0 (socket says 172.21.15.176:55236) 2024-02-19T22:07:36.467 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.465+0000 7f8c7a59c700 1 -- 172.21.15.176:0/2575396238 learned_addr learned my addr 172.21.15.176:0/2575396238 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:36.467 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.465+0000 7f8c7a59c700 1 -- 172.21.15.176:0/2575396238 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c0ff0e0 con 0x7f8c7c0703d0 2024-02-19T22:07:36.467 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.465+0000 7f8c7a59c700 1 --2- 172.21.15.176:0/2575396238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c7c0703d0 0x7f8c7c0feab0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f8c7000b5f0 tx=0x7f8c70006ee0).ready entity=mon.0 client_cookie=16b7a13388924c57 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:36.467 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.465+0000 7f8c7959a700 1 -- 172.21.15.176:0/2575396238 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c7000d600 con 0x7f8c7c0703d0 2024-02-19T22:07:36.467 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.465+0000 7f8c7959a700 1 -- 172.21.15.176:0/2575396238 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7000d760 con 0x7f8c7c0703d0 2024-02-19T22:07:36.467 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.465+0000 7f8c7959a700 1 -- 172.21.15.176:0/2575396238 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c7000da30 con 0x7f8c7c0703d0 2024-02-19T22:07:36.468 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.466+0000 7f8c80f1b700 1 -- 172.21.15.176:0/2575396238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c7c0703d0 msgr2=0x7f8c7c0feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:36.469 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.466+0000 7f8c80f1b700 1 --2- 172.21.15.176:0/2575396238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c7c0703d0 0x7f8c7c0feab0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f8c7000b5f0 tx=0x7f8c70006ee0).stop 2024-02-19T22:07:36.469 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.466+0000 7f8c80f1b700 1 -- 172.21.15.176:0/2575396238 shutdown_connections 2024-02-19T22:07:36.469 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.466+0000 7f8c80f1b700 1 --2- 172.21.15.176:0/2575396238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c7c0703d0 0x7f8c7c0feab0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:36.469 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.466+0000 7f8c80f1b700 1 -- 172.21.15.176:0/2575396238 >> 172.21.15.176:0/2575396238 conn(0x7f8c7c0f6530 msgr2=0x7f8c7c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:36.469 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.466+0000 7f8c80f1b700 1 -- 172.21.15.176:0/2575396238 shutdown_connections 2024-02-19T22:07:36.469 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.466+0000 7f8c80f1b700 1 -- 172.21.15.176:0/2575396238 wait complete. 2024-02-19T22:07:36.470 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.467+0000 7f8c80f1b700 1 Processor -- start 2024-02-19T22:07:36.470 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.467+0000 7f8c80f1b700 1 -- start start 2024-02-19T22:07:36.470 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.467+0000 7f8c80f1b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c7c0703d0 0x7f8c7c0fc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:36.470 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.467+0000 7f8c80f1b700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c0fa9b0 con 0x7f8c7c0703d0 2024-02-19T22:07:36.470 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.467+0000 7f8c7a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c7c0703d0 0x7f8c7c0fc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:36.470 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.467+0000 7f8c7a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c7c0703d0 0x7f8c7c0fc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:55240/0 (socket says 172.21.15.176:55240) 2024-02-19T22:07:36.470 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.467+0000 7f8c7a59c700 1 -- 172.21.15.176:0/1321544195 learned_addr learned my addr 172.21.15.176:0/1321544195 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:36.470 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.468+0000 7f8c7a59c700 1 -- 172.21.15.176:0/1321544195 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7000b200 con 0x7f8c7c0703d0 2024-02-19T22:07:36.470 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.468+0000 7f8c7a59c700 1 --2- 172.21.15.176:0/1321544195 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c7c0703d0 0x7f8c7c0fc530 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f8c70006c70 tx=0x7f8c70016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:36.470 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.468+0000 7f8c6affd700 1 -- 172.21.15.176:0/1321544195 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c70011b40 con 0x7f8c7c0703d0 2024-02-19T22:07:36.470 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.468+0000 7f8c6affd700 1 -- 172.21.15.176:0/1321544195 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c70011ca0 con 0x7f8c7c0703d0 2024-02-19T22:07:36.470 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.468+0000 7f8c6affd700 1 -- 172.21.15.176:0/1321544195 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c7001d430 con 0x7f8c7c0703d0 2024-02-19T22:07:36.470 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.468+0000 7f8c80f1b700 1 -- 172.21.15.176:0/1321544195 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7c0fab50 con 0x7f8c7c0703d0 2024-02-19T22:07:36.471 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.468+0000 7f8c80f1b700 1 -- 172.21.15.176:0/1321544195 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c0fafe0 con 0x7f8c7c0703d0 2024-02-19T22:07:36.471 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.468+0000 7f8c6affd700 1 -- 172.21.15.176:0/1321544195 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f8c7000f070 con 0x7f8c7c0703d0 2024-02-19T22:07:36.471 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.468+0000 7f8c6affd700 1 -- 172.21.15.176:0/1321544195 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f8c70027460 con 0x7f8c7c0703d0 2024-02-19T22:07:36.471 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.469+0000 7f8c80f1b700 1 -- 172.21.15.176:0/1321544195 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7c045ad0 con 0x7f8c7c0703d0 2024-02-19T22:07:36.472 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.471+0000 7f8c6affd700 1 -- 172.21.15.176:0/1321544195 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8c70016070 con 0x7f8c7c0703d0 2024-02-19T22:07:36.536 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.535+0000 7f8c80f1b700 1 -- 172.21.15.176:0/1321544195 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f8c7c0ff390 con 0x7f8c7c0703d0 2024-02-19T22:07:36.539 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.538+0000 7f8c6affd700 1 -- 172.21.15.176:0/1321544195 <== mon.0 v2:172.21.15.176:3300/0 7 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f8c7001d660 con 0x7f8c7c0703d0 2024-02-19T22:07:36.540 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.538+0000 7f8c6affd700 1 -- 172.21.15.176:0/1321544195 <== mon.0 v2:172.21.15.176:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+491 (secure 0 0 0) 0x7f8c70036020 con 0x7f8c7c0703d0 2024-02-19T22:07:36.540 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.540 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: [global] 2024-02-19T22:07:36.540 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: fsid = 36a8f1ec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:07:36.540 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-19T22:07:36.540 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.176:3300,v1:172.21.15.176:6789] 2024-02-19T22:07:36.540 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-19T22:07:36.540 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-19T22:07:36.540 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-19T22:07:36.541 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-19T22:07:36.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: [mgr] 2024-02-19T22:07:36.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-19T22:07:36.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: [osd] 2024-02-19T22:07:36.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-19T22:07:36.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-19T22:07:36.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-19T22:07:36.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-19T22:07:36.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-02-19T22:07:36.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-19T22:07:36.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.539+0000 7f8c80f1b700 1 -- 172.21.15.176:0/1321544195 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c7c0703d0 msgr2=0x7f8c7c0fc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:36.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.539+0000 7f8c80f1b700 1 --2- 172.21.15.176:0/1321544195 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c7c0703d0 0x7f8c7c0fc530 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f8c70006c70 tx=0x7f8c70016000).stop 2024-02-19T22:07:36.543 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.539+0000 7f8c80f1b700 1 -- 172.21.15.176:0/1321544195 shutdown_connections 2024-02-19T22:07:36.543 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.539+0000 7f8c80f1b700 1 --2- 172.21.15.176:0/1321544195 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8c7c0703d0 0x7f8c7c0fc530 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:36.543 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.539+0000 7f8c80f1b700 1 -- 172.21.15.176:0/1321544195 >> 172.21.15.176:0/1321544195 conn(0x7f8c7c0f6530 msgr2=0x7f8c7c0f81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:36.543 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.539+0000 7f8c80f1b700 1 -- 172.21.15.176:0/1321544195 shutdown_connections 2024-02-19T22:07:36.543 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:36.539+0000 7f8c80f1b700 1 -- 172.21.15.176:0/1321544195 wait complete. 2024-02-19T22:07:36.824 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[37699]: audit 2024-02-19T22:07:36.537026+0000 mon.smithi176 (mon.0) 8 : audit [INF] from='client.? 172.21.15.176:0/1321544195' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-19T22:07:36.824 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[37699]: audit 2024-02-19T22:07:36.539990+0000 mon.smithi176 (mon.0) 9 : audit [INF] from='client.? 172.21.15.176:0/1321544195' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-19T22:07:37.077 INFO:teuthology.orchestra.run.smithi176.stderr:Generating new minimal ceph.conf... 2024-02-19T22:07:37.077 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z -v /var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/mon.smithi176:/var/lib/ceph/mon/ceph-smithi176:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi176/config 2024-02-19T22:07:37.578 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.575+0000 7fbe7e87d700 1 Processor -- start 2024-02-19T22:07:37.578 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.575+0000 7fbe7e87d700 1 -- start start 2024-02-19T22:07:37.578 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.575+0000 7fbe7e87d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe78070390 0x7fbe78100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:37.578 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.575+0000 7fbe7e87d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe780fc8e0 con 0x7fbe78070390 2024-02-19T22:07:37.579 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.576+0000 7fbe77fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe78070390 0x7fbe78100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:37.579 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.576+0000 7fbe77fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe78070390 0x7fbe78100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:55248/0 (socket says 172.21.15.176:55248) 2024-02-19T22:07:37.579 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.576+0000 7fbe77fff700 1 -- 172.21.15.176:0/714151395 learned_addr learned my addr 172.21.15.176:0/714151395 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:37.579 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.576+0000 7fbe77fff700 1 -- 172.21.15.176:0/714151395 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe781013e0 con 0x7fbe78070390 2024-02-19T22:07:37.579 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.576+0000 7fbe77fff700 1 --2- 172.21.15.176:0/714151395 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe78070390 0x7fbe78100ef0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fbe6800b5f0 tx=0x7fbe68006ee0).ready entity=mon.0 client_cookie=68242b999cc5d9f0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:37.579 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.576+0000 7fbe76ffd700 1 -- 172.21.15.176:0/714151395 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe6800d600 con 0x7fbe78070390 2024-02-19T22:07:37.579 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.576+0000 7fbe76ffd700 1 -- 172.21.15.176:0/714151395 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbe6800dbf0 con 0x7fbe78070390 2024-02-19T22:07:37.579 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.576+0000 7fbe76ffd700 1 -- 172.21.15.176:0/714151395 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe68011cb0 con 0x7fbe78070390 2024-02-19T22:07:37.579 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.577+0000 7fbe7e87d700 1 -- 172.21.15.176:0/714151395 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe78070390 msgr2=0x7fbe78100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:37.579 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.577+0000 7fbe7e87d700 1 --2- 172.21.15.176:0/714151395 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe78070390 0x7fbe78100ef0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fbe6800b5f0 tx=0x7fbe68006ee0).stop 2024-02-19T22:07:37.579 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.577+0000 7fbe7e87d700 1 -- 172.21.15.176:0/714151395 shutdown_connections 2024-02-19T22:07:37.580 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.577+0000 7fbe7e87d700 1 --2- 172.21.15.176:0/714151395 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe78070390 0x7fbe78100ef0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:37.580 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.577+0000 7fbe7e87d700 1 -- 172.21.15.176:0/714151395 >> 172.21.15.176:0/714151395 conn(0x7fbe780f6530 msgr2=0x7fbe780f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:37.580 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.577+0000 7fbe7e87d700 1 -- 172.21.15.176:0/714151395 shutdown_connections 2024-02-19T22:07:37.582 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.577+0000 7fbe7e87d700 1 -- 172.21.15.176:0/714151395 wait complete. 2024-02-19T22:07:37.582 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.577+0000 7fbe7e87d700 1 Processor -- start 2024-02-19T22:07:37.582 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.578+0000 7fbe7e87d700 1 -- start start 2024-02-19T22:07:37.582 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.578+0000 7fbe7e87d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe78070390 0x7fbe7810cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:37.582 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.578+0000 7fbe7e87d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7810d2b0 con 0x7fbe78070390 2024-02-19T22:07:37.582 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.578+0000 7fbe77fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe78070390 0x7fbe7810cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:37.582 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.578+0000 7fbe77fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe78070390 0x7fbe7810cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:55264/0 (socket says 172.21.15.176:55264) 2024-02-19T22:07:37.582 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.578+0000 7fbe77fff700 1 -- 172.21.15.176:0/2918331978 learned_addr learned my addr 172.21.15.176:0/2918331978 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:37.582 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.578+0000 7fbe77fff700 1 -- 172.21.15.176:0/2918331978 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6800b200 con 0x7fbe78070390 2024-02-19T22:07:37.582 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.578+0000 7fbe77fff700 1 --2- 172.21.15.176:0/2918331978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe78070390 0x7fbe7810cdc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbe6800df90 tx=0x7fbe68006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:37.582 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.578+0000 7fbe74ff9700 1 -- 172.21.15.176:0/2918331978 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe6801b410 con 0x7fbe78070390 2024-02-19T22:07:37.583 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.578+0000 7fbe74ff9700 1 -- 172.21.15.176:0/2918331978 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbe6801ba00 con 0x7fbe78070390 2024-02-19T22:07:37.583 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.578+0000 7fbe74ff9700 1 -- 172.21.15.176:0/2918331978 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe6801aa30 con 0x7fbe78070390 2024-02-19T22:07:37.583 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.578+0000 7fbe7e87d700 1 -- 172.21.15.176:0/2918331978 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe7810d4b0 con 0x7fbe78070390 2024-02-19T22:07:37.583 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.579+0000 7fbe7e87d700 1 -- 172.21.15.176:0/2918331978 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7810e050 con 0x7fbe78070390 2024-02-19T22:07:37.583 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.579+0000 7fbe74ff9700 1 -- 172.21.15.176:0/2918331978 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fbe6800f070 con 0x7fbe78070390 2024-02-19T22:07:37.583 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.579+0000 7fbe74ff9700 1 -- 172.21.15.176:0/2918331978 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fbe68025c30 con 0x7fbe78070390 2024-02-19T22:07:37.583 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.580+0000 7fbe7e87d700 1 -- 172.21.15.176:0/2918331978 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe78059100 con 0x7fbe78070390 2024-02-19T22:07:37.583 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.582+0000 7fbe74ff9700 1 -- 172.21.15.176:0/2918331978 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fbe68016070 con 0x7fbe78070390 2024-02-19T22:07:37.647 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.646+0000 7fbe7e87d700 1 -- 172.21.15.176:0/2918331978 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fbe78102000 con 0x7fbe78070390 2024-02-19T22:07:37.647 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.646+0000 7fbe74ff9700 1 -- 172.21.15.176:0/2918331978 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fbe68025460 con 0x7fbe78070390 2024-02-19T22:07:37.648 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.647+0000 7fbe7e87d700 1 -- 172.21.15.176:0/2918331978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe78070390 msgr2=0x7fbe7810cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:37.649 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.647+0000 7fbe7e87d700 1 --2- 172.21.15.176:0/2918331978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe78070390 0x7fbe7810cdc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbe6800df90 tx=0x7fbe68006ee0).stop 2024-02-19T22:07:37.649 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.647+0000 7fbe7e87d700 1 -- 172.21.15.176:0/2918331978 shutdown_connections 2024-02-19T22:07:37.649 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.647+0000 7fbe7e87d700 1 --2- 172.21.15.176:0/2918331978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbe78070390 0x7fbe7810cdc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:37.649 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.647+0000 7fbe7e87d700 1 -- 172.21.15.176:0/2918331978 >> 172.21.15.176:0/2918331978 conn(0x7fbe780f6530 msgr2=0x7fbe780f7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:37.649 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.647+0000 7fbe7e87d700 1 -- 172.21.15.176:0/2918331978 shutdown_connections 2024-02-19T22:07:37.649 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:37.647+0000 7fbe7e87d700 1 -- 172.21.15.176:0/2918331978 wait complete. 2024-02-19T22:07:37.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[37699]: audit 2024-02-19T22:07:37.648279+0000 mon.smithi176 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.176:0/2918331978' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T22:07:38.269 INFO:teuthology.orchestra.run.smithi176.stderr:Restarting the monitor... 2024-02-19T22:07:38.269 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl restart ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi176 2024-02-19T22:07:38.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:38 smithi176 systemd[1]: Stopping Ceph mon.smithi176 for 36a8f1ec-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:07:38.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:38 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[37699]: debug 2024-02-19T22:07:38.406+0000 7f655c167700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi176 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T22:07:38.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:38 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[37699]: debug 2024-02-19T22:07:38.406+0000 7f655c167700 -1 mon.smithi176@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T22:07:39.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:38 smithi176 podman[38025]: ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176 2024-02-19T22:07:39.573 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 systemd[1]: ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi176.service: Succeeded. 2024-02-19T22:07:39.574 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 systemd[1]: Stopped Ceph mon.smithi176 for 36a8f1ec-cf73-11ee-95bb-87774f69a715. 2024-02-19T22:07:39.574 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 systemd[1]: Starting Ceph mon.smithi176 for 36a8f1ec-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:07:39.766 INFO:teuthology.orchestra.run.smithi176.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-19T22:07:39.766 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-02-19T22:07:39.975 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.697+0000 7fc4a6f5a700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T22:07:39.975 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.697+0000 7fc4a6f5a700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-02-19T22:07:39.975 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.697+0000 7fc4a6f5a700 0 pidfile_write: ignore empty --pid-file 2024-02-19T22:07:39.975 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.701+0000 7fc4a6f5a700 0 load: jerasure load: lrc load: isa 2024-02-19T22:07:39.975 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T22:07:39.975 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.975 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T22:07:39.975 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Compile date May 13 2021 2024-02-19T22:07:39.975 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: DB SUMMARY 2024-02-19T22:07:39.976 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.976 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: CURRENT file: CURRENT 2024-02-19T22:07:39.976 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.976 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-19T22:07:39.976 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.976 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-19T22:07:39.976 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.976 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi176/store.db dir, Total Num: 1, files: 000004.sst 2024-02-19T22:07:39.976 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.976 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi176/store.db: 000006.log size: 98740 ; 2024-02-19T22:07:39.976 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.976 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.error_if_exists: 0 2024-02-19T22:07:39.976 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.create_if_missing: 0 2024-02-19T22:07:39.976 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T22:07:39.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.env: 0x55b0200cd1c0 2024-02-19T22:07:39.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.fs: Posix File System 2024-02-19T22:07:39.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.info_log: 0x55b021ffb960 2024-02-19T22:07:39.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T22:07:39.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.statistics: (nil) 2024-02-19T22:07:39.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.use_fsync: 0 2024-02-19T22:07:39.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T22:07:39.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T22:07:39.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T22:07:39.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T22:07:39.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T22:07:39.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T22:07:39.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T22:07:39.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.db_log_dir: 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi176/store.db 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T22:07:39.978 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T22:07:39.979 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.write_buffer_manager: 0x55b022001fb0 2024-02-19T22:07:39.979 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T22:07:39.979 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T22:07:39.979 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T22:07:39.979 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T22:07:39.979 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T22:07:39.979 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T22:07:39.979 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T22:07:39.979 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T22:07:39.979 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T22:07:39.979 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.unordered_write: 0 2024-02-19T22:07:39.979 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T22:07:39.979 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T22:07:39.979 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T22:07:39.980 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T22:07:39.980 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.row_cache: None 2024-02-19T22:07:39.980 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.wal_filter: None 2024-02-19T22:07:39.980 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T22:07:39.980 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T22:07:39.980 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T22:07:39.980 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.two_write_queues: 0 2024-02-19T22:07:39.980 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T22:07:39.980 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.atomic_flush: 0 2024-02-19T22:07:39.980 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T22:07:39.980 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T22:07:39.980 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T22:07:39.980 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T22:07:39.981 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T22:07:39.981 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T22:07:39.981 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T22:07:39.981 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T22:07:39.981 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T22:07:39.981 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T22:07:39.981 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T22:07:39.981 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T22:07:39.981 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T22:07:39.981 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T22:07:39.981 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T22:07:39.981 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_open_files: -1 2024-02-19T22:07:39.981 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T22:07:39.981 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T22:07:39.982 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T22:07:39.982 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T22:07:39.982 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Compression algorithms supported: 2024-02-19T22:07:39.982 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T22:07:39.982 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: kZSTD supported: 0 2024-02-19T22:07:39.982 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: kXpressCompression supported: 0 2024-02-19T22:07:39.982 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T22:07:39.982 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T22:07:39.982 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T22:07:39.982 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: kZlibCompression supported: 1 2024-02-19T22:07:39.982 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T22:07:39.982 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T22:07:39.982 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi176/store.db/MANIFEST-000005 2024-02-19T22:07:39.982 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.983 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T22:07:39.983 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.983 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T22:07:39.983 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.merge_operator: 2024-02-19T22:07:39.983 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compaction_filter: None 2024-02-19T22:07:39.983 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T22:07:39.983 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T22:07:39.983 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T22:07:39.983 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b021f13cb0) 2024-02-19T22:07:39.983 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cache_index_and_filter_blocks: 1 2024-02-19T22:07:39.983 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T22:07:39.983 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T22:07:39.983 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: pin_top_level_index_and_filter: 1 2024-02-19T22:07:39.983 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: index_type: 0 2024-02-19T22:07:39.984 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: data_block_index_type: 0 2024-02-19T22:07:39.984 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: index_shortening: 1 2024-02-19T22:07:39.984 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: data_block_hash_table_util_ratio: 0.750000 2024-02-19T22:07:39.984 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: hash_index_allow_collision: 1 2024-02-19T22:07:39.984 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: checksum: 1 2024-02-19T22:07:39.984 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: no_block_cache: 0 2024-02-19T22:07:39.984 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: block_cache: 0x55b021f48f10 2024-02-19T22:07:39.984 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: block_cache_name: BinnedLRUCache 2024-02-19T22:07:39.984 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: block_cache_options: 2024-02-19T22:07:39.984 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: capacity : 536870912 2024-02-19T22:07:39.984 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: num_shard_bits : 4 2024-02-19T22:07:39.984 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: strict_capacity_limit : 0 2024-02-19T22:07:39.984 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: high_pri_pool_ratio: 0.000 2024-02-19T22:07:39.985 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: block_cache_compressed: (nil) 2024-02-19T22:07:39.985 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: persistent_cache: (nil) 2024-02-19T22:07:39.985 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: block_size: 4096 2024-02-19T22:07:39.985 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: block_size_deviation: 10 2024-02-19T22:07:39.985 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: block_restart_interval: 16 2024-02-19T22:07:39.985 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: index_block_restart_interval: 1 2024-02-19T22:07:39.985 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: metadata_block_size: 4096 2024-02-19T22:07:39.985 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: partition_filters: 0 2024-02-19T22:07:39.985 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: use_delta_encoding: 1 2024-02-19T22:07:39.985 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T22:07:39.985 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: whole_key_filtering: 1 2024-02-19T22:07:39.985 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: verify_compression: 0 2024-02-19T22:07:39.985 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: read_amp_bytes_per_bit: 0 2024-02-19T22:07:39.985 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: format_version: 2 2024-02-19T22:07:39.986 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: enable_index_compression: 1 2024-02-19T22:07:39.986 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: block_align: 0 2024-02-19T22:07:39.986 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.986 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T22:07:39.986 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T22:07:39.986 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compression: NoCompression 2024-02-19T22:07:39.986 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T22:07:39.986 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T22:07:39.986 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T22:07:39.986 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.num_levels: 7 2024-02-19T22:07:39.986 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T22:07:39.986 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T22:07:39.986 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T22:07:39.987 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T22:07:39.987 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T22:07:39.987 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T22:07:39.987 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T22:07:39.987 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:07:39.987 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T22:07:39.987 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T22:07:39.987 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T22:07:39.987 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T22:07:39.987 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T22:07:39.987 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:07:39.987 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T22:07:39.987 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T22:07:39.987 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T22:07:39.988 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T22:07:39.988 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T22:07:39.988 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T22:07:39.988 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T22:07:39.988 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T22:07:39.988 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T22:07:39.988 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T22:07:39.988 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T22:07:39.988 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T22:07:39.988 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T22:07:39.988 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T22:07:39.988 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T22:07:39.988 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T22:07:39.988 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T22:07:39.989 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T22:07:39.989 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T22:07:39.989 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T22:07:39.989 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T22:07:39.989 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T22:07:39.989 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T22:07:39.989 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T22:07:39.989 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T22:07:39.989 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T22:07:39.989 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T22:07:39.989 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T22:07:39.989 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T22:07:39.989 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T22:07:39.989 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T22:07:39.990 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T22:07:39.990 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T22:07:39.990 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.table_properties_collectors: 2024-02-19T22:07:39.990 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T22:07:39.990 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T22:07:39.990 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T22:07:39.990 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T22:07:39.990 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T22:07:39.990 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.bloom_locality: 0 2024-02-19T22:07:39.990 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T22:07:39.990 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T22:07:39.990 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T22:07:39.990 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T22:07:39.990 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T22:07:39.991 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.ttl: 2592000 2024-02-19T22:07:39.991 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T22:07:39.991 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi176/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T22:07:39.991 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.991 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-19T22:07:39.991 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.991 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708380459704378, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-19T22:07:39.991 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.702+0000 7fc4a6f5a700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-19T22:07:39.991 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.703+0000 7fc4a6f5a700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-19T22:07:39.991 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.703+0000 7fc4a6f5a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708380459705225, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 94314, "table_properties": {"data_size": 92402, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12563, "raw_average_key_size": 49, "raw_value_size": 85433, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 254, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708380459, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-19T22:07:39.991 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.703+0000 7fc4a6f5a700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-19T22:07:39.991 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.991 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.704+0000 7fc4a6f5a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708380459706240, "job": 1, "event": "recovery_finished"} 2024-02-19T22:07:39.991 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.705+0000 7fc4a6f5a700 4 rocksdb: DB pointer 0x55b02200d800 2024-02-19T22:07:39.992 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.705+0000 7fc48dd72700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T22:07:39.992 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.705+0000 7fc48dd72700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T22:07:39.992 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: ** DB Stats ** 2024-02-19T22:07:39.992 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:07:39.992 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T22:07:39.992 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T22:07:39.992 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:07:39.993 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T22:07:39.993 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T22:07:39.993 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:07:39.993 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.993 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: ** Compaction Stats [default] ** 2024-02-19T22:07:39.993 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:07:39.993 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:07:39.993 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: L0 2/0 93.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.6 0.00 0.00 1 0.001 0 0 2024-02-19T22:07:39.994 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Sum 2/0 93.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.6 0.00 0.00 1 0.001 0 0 2024-02-19T22:07:39.994 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.6 0.00 0.00 1 0.001 0 0 2024-02-19T22:07:39.994 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.994 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: ** Compaction Stats [default] ** 2024-02-19T22:07:39.994 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:07:39.994 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:07:39.994 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.6 0.00 0.00 1 0.001 0 0 2024-02-19T22:07:39.994 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:07:39.994 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:07:39.994 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:07:39.994 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:07:39.994 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:07:39.994 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:07:39.994 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Cumulative compaction: 0.00 GB write, 27.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:07:39.995 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Interval compaction: 0.00 GB write, 27.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:07:39.995 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:07:39.995 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.995 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:07:39.995 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.995 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: ** Compaction Stats [default] ** 2024-02-19T22:07:39.995 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:07:39.995 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:07:39.995 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: L0 2/0 93.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.6 0.00 0.00 1 0.001 0 0 2024-02-19T22:07:39.995 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Sum 2/0 93.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.6 0.00 0.00 1 0.001 0 0 2024-02-19T22:07:39.995 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:07:39.995 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.996 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: ** Compaction Stats [default] ** 2024-02-19T22:07:39.996 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:07:39.996 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:07:39.996 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.6 0.00 0.00 1 0.001 0 0 2024-02-19T22:07:39.996 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:07:39.996 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:07:39.996 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:07:39.996 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:07:39.996 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:07:39.996 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:07:39.996 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Cumulative compaction: 0.00 GB write, 27.10 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:07:39.996 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:07:39.996 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:07:39.996 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.706+0000 7fc4a6f5a700 0 starting mon.smithi176 rank 0 at public addrs [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] at bind addrs [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi176 fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.706+0000 7fc4a6f5a700 1 mon.smithi176@-1(???) e1 preinit fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.707+0000 7fc4a6f5a700 0 mon.smithi176@-1(???).mds e1 new map 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.707+0000 7fc4a6f5a700 0 mon.smithi176@-1(???).mds e1 print_map 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: e1 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: legacy client fscid: -1 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: No filesystems configured 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.707+0000 7fc4a6f5a700 0 mon.smithi176@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-19T22:07:39.997 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.707+0000 7fc4a6f5a700 0 mon.smithi176@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T22:07:39.998 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.707+0000 7fc4a6f5a700 0 mon.smithi176@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T22:07:39.998 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.707+0000 7fc4a6f5a700 0 mon.smithi176@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T22:07:39.998 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:37.648279+0000 mon.smithi176 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.176:0/2918331978' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T22:07:39.998 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.707+0000 7fc4a6f5a700 1 mon.smithi176@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-19T22:07:39.998 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.708+0000 7fc4a6f5a700 20 expand_channel_meta expand map: {default=false} 2024-02-19T22:07:39.998 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.708+0000 7fc4a6f5a700 20 expand_channel_meta from 'false' to 'false' 2024-02-19T22:07:39.998 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.708+0000 7fc4a6f5a700 20 expand_channel_meta expanded map: {default=false} 2024-02-19T22:07:39.998 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.708+0000 7fc4a6f5a700 20 expand_channel_meta expand map: {default=info} 2024-02-19T22:07:39.998 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.708+0000 7fc4a6f5a700 20 expand_channel_meta from 'info' to 'info' 2024-02-19T22:07:39.998 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.708+0000 7fc4a6f5a700 20 expand_channel_meta expanded map: {default=info} 2024-02-19T22:07:39.998 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.708+0000 7fc4a6f5a700 20 expand_channel_meta expand map: {default=daemon} 2024-02-19T22:07:39.998 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.708+0000 7fc4a6f5a700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-02-19T22:07:39.998 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.708+0000 7fc4a6f5a700 20 expand_channel_meta expanded map: {default=daemon} 2024-02-19T22:07:39.999 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.708+0000 7fc4a6f5a700 20 expand_channel_meta expand map: {default=debug} 2024-02-19T22:07:39.999 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.708+0000 7fc4a6f5a700 20 expand_channel_meta from 'debug' to 'debug' 2024-02-19T22:07:39.999 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:07:39.708+0000 7fc4a6f5a700 20 expand_channel_meta expanded map: {default=debug} 2024-02-19T22:07:39.999 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 bash[38094]: 3e9cd3afa43f5e7088ed430a548e762fc7ffea6588e1b6a4c431a032e372d133 2024-02-19T22:07:39.999 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 systemd[1]: Started Ceph mon.smithi176 for 36a8f1ec-cf73-11ee-95bb-87774f69a715. 2024-02-19T22:07:39.999 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:39.724235+0000 mon.smithi176 (mon.0) 1 : cluster [INF] mon.smithi176 is new leader, mons smithi176 in quorum (ranks 0) 2024-02-19T22:07:39.999 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:39.724295+0000 mon.smithi176 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi176=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} 2024-02-19T22:07:39.999 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:39.724330+0000 mon.smithi176 (mon.0) 3 : cluster [DBG] fsmap 2024-02-19T22:07:39.999 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:39.724353+0000 mon.smithi176 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-19T22:07:39.999 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:39.724701+0000 mon.smithi176 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-19T22:07:40.297 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.296+0000 7f27460fd700 1 Processor -- start 2024-02-19T22:07:40.297 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.296+0000 7f27460fd700 1 -- start start 2024-02-19T22:07:40.298 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.296+0000 7f27460fd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27400fab30 0x7f27400faf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:40.298 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.296+0000 7f27460fd700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27400fb440 con 0x7f27400fab30 2024-02-19T22:07:40.299 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.297+0000 7f27450fb700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27400fab30 0x7f27400faf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:40.299 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.297+0000 7f27450fb700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27400fab30 0x7f27400faf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:55270/0 (socket says 172.21.15.176:55270) 2024-02-19T22:07:40.299 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.297+0000 7f27450fb700 1 -- 172.21.15.176:0/3335987089 learned_addr learned my addr 172.21.15.176:0/3335987089 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:40.299 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.297+0000 7f27450fb700 1 -- 172.21.15.176:0/3335987089 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27400fb580 con 0x7f27400fab30 2024-02-19T22:07:40.299 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.297+0000 7f27450fb700 1 --2- 172.21.15.176:0/3335987089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27400fab30 0x7f27400faf50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f273000b5f0 tx=0x7f2730006ee0).ready entity=mon.0 client_cookie=51a09428c43d292e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:40.300 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.298+0000 7f272ffff700 1 -- 172.21.15.176:0/3335987089 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f273000d600 con 0x7f27400fab30 2024-02-19T22:07:40.300 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.298+0000 7f272ffff700 1 -- 172.21.15.176:0/3335987089 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f273000dbf0 con 0x7f27400fab30 2024-02-19T22:07:40.300 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.298+0000 7f272ffff700 1 -- 172.21.15.176:0/3335987089 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2730011cb0 con 0x7f27400fab30 2024-02-19T22:07:40.300 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.298+0000 7f27460fd700 1 -- 172.21.15.176:0/3335987089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27400fab30 msgr2=0x7f27400faf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:40.300 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.298+0000 7f27460fd700 1 --2- 172.21.15.176:0/3335987089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27400fab30 0x7f27400faf50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f273000b5f0 tx=0x7f2730006ee0).stop 2024-02-19T22:07:40.300 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.298+0000 7f27460fd700 1 -- 172.21.15.176:0/3335987089 shutdown_connections 2024-02-19T22:07:40.300 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.298+0000 7f27460fd700 1 --2- 172.21.15.176:0/3335987089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27400fab30 0x7f27400faf50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:40.300 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.298+0000 7f27460fd700 1 -- 172.21.15.176:0/3335987089 >> 172.21.15.176:0/3335987089 conn(0x7f27400f6390 msgr2=0x7f27400f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:40.301 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.299+0000 7f27460fd700 1 -- 172.21.15.176:0/3335987089 shutdown_connections 2024-02-19T22:07:40.301 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.299+0000 7f27460fd700 1 -- 172.21.15.176:0/3335987089 wait complete. 2024-02-19T22:07:40.301 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.299+0000 7f27460fd700 1 Processor -- start 2024-02-19T22:07:40.301 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.299+0000 7f27460fd700 1 -- start start 2024-02-19T22:07:40.302 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.300+0000 7f27460fd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27400fab30 0x7f274010a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:40.302 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.300+0000 7f27460fd700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274010aec0 con 0x7f27400fab30 2024-02-19T22:07:40.302 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.300+0000 7f27450fb700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27400fab30 0x7f274010a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:40.302 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.300+0000 7f27450fb700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27400fab30 0x7f274010a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:55278/0 (socket says 172.21.15.176:55278) 2024-02-19T22:07:40.302 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.300+0000 7f27450fb700 1 -- 172.21.15.176:0/381121253 learned_addr learned my addr 172.21.15.176:0/381121253 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:40.302 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.300+0000 7f27450fb700 1 -- 172.21.15.176:0/381121253 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273000b200 con 0x7f27400fab30 2024-02-19T22:07:40.302 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.300+0000 7f27450fb700 1 --2- 172.21.15.176:0/381121253 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27400fab30 0x7f274010a9d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2730002f40 tx=0x7f2730006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:40.302 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.300+0000 7f272dffb700 1 -- 172.21.15.176:0/381121253 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f273001b410 con 0x7f27400fab30 2024-02-19T22:07:40.303 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.300+0000 7f272dffb700 1 -- 172.21.15.176:0/381121253 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f273001ba00 con 0x7f27400fab30 2024-02-19T22:07:40.303 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.300+0000 7f272dffb700 1 -- 172.21.15.176:0/381121253 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f273001a900 con 0x7f27400fab30 2024-02-19T22:07:40.303 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.300+0000 7f27460fd700 1 -- 172.21.15.176:0/381121253 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f274010b0c0 con 0x7f27400fab30 2024-02-19T22:07:40.303 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.300+0000 7f27460fd700 1 -- 172.21.15.176:0/381121253 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f274010b550 con 0x7f27400fab30 2024-02-19T22:07:40.303 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.301+0000 7f272dffb700 1 -- 172.21.15.176:0/381121253 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f273000f070 con 0x7f27400fab30 2024-02-19T22:07:40.303 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.301+0000 7f272dffb700 1 -- 172.21.15.176:0/381121253 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2730025c30 con 0x7f27400fab30 2024-02-19T22:07:40.303 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.301+0000 7f27460fd700 1 -- 172.21.15.176:0/381121253 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2740045ad0 con 0x7f27400fab30 2024-02-19T22:07:40.304 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.304+0000 7f272dffb700 1 -- 172.21.15.176:0/381121253 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2730016070 con 0x7f27400fab30 2024-02-19T22:07:40.367 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.366+0000 7f27460fd700 1 -- 172.21.15.176:0/381121253 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f27400fc900 con 0x7f27400fab30 2024-02-19T22:07:40.369 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.368+0000 7f272dffb700 1 -- 172.21.15.176:0/381121253 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2730025460 con 0x7f27400fab30 2024-02-19T22:07:40.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.369+0000 7f27460fd700 1 -- 172.21.15.176:0/381121253 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27400fab30 msgr2=0x7f274010a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:40.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.369+0000 7f27460fd700 1 --2- 172.21.15.176:0/381121253 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27400fab30 0x7f274010a9d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2730002f40 tx=0x7f2730006ee0).stop 2024-02-19T22:07:40.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.369+0000 7f27460fd700 1 -- 172.21.15.176:0/381121253 shutdown_connections 2024-02-19T22:07:40.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.369+0000 7f27460fd700 1 --2- 172.21.15.176:0/381121253 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27400fab30 0x7f274010a9d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:40.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.369+0000 7f27460fd700 1 -- 172.21.15.176:0/381121253 >> 172.21.15.176:0/381121253 conn(0x7f27400f6390 msgr2=0x7f2740070010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:40.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.369+0000 7f27460fd700 1 -- 172.21.15.176:0/381121253 shutdown_connections 2024-02-19T22:07:40.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:40.369+0000 7f27460fd700 1 -- 172.21.15.176:0/381121253 wait complete. 2024-02-19T22:07:41.005 INFO:teuthology.orchestra.run.smithi176.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-19T22:07:41.005 INFO:teuthology.orchestra.run.smithi176.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-19T22:07:41.005 INFO:teuthology.orchestra.run.smithi176.stderr:Creating mgr... 2024-02-19T22:07:41.005 INFO:teuthology.orchestra.run.smithi176.stderr:Verifying port 9283 ... 2024-02-19T22:07:41.007 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl daemon-reload 2024-02-19T22:07:41.102 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl stop ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mgr.smithi176.faoogw 2024-02-19T22:07:41.125 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl reset-failed ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mgr.smithi176.faoogw 2024-02-19T22:07:41.130 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: Failed to reset failed state of unit ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mgr.smithi176.faoogw.service: Unit ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mgr.smithi176.faoogw.service not loaded. 2024-02-19T22:07:41.131 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl enable ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mgr.smithi176.faoogw 2024-02-19T22:07:41.136 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: Created symlink /etc/systemd/system/ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715.target.wants/ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mgr.smithi176.faoogw.service → /etc/systemd/system/ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@.service. 2024-02-19T22:07:41.221 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl start ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mgr.smithi176.faoogw 2024-02-19T22:07:41.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:40.370195+0000 mon.smithi176 (mon.0) 6 : audit [INF] from='client.? 172.21.15.176:0/381121253' entity='client.admin' 2024-02-19T22:07:41.819 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:07:41.827 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: disabled 2024-02-19T22:07:41.827 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:07:41.835 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: inactive 2024-02-19T22:07:41.836 INFO:teuthology.orchestra.run.smithi176.stderr:firewalld.service is not enabled 2024-02-19T22:07:41.836 INFO:teuthology.orchestra.run.smithi176.stderr:Not possible to enable service . firewalld.service is not available 2024-02-19T22:07:41.836 INFO:teuthology.orchestra.run.smithi176.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-19T22:07:41.836 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:07:41.844 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: disabled 2024-02-19T22:07:41.844 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:07:41.851 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: inactive 2024-02-19T22:07:41.853 INFO:teuthology.orchestra.run.smithi176.stderr:firewalld.service is not enabled 2024-02-19T22:07:41.853 INFO:teuthology.orchestra.run.smithi176.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-19T22:07:41.853 INFO:teuthology.orchestra.run.smithi176.stderr:Waiting for mgr to start... 2024-02-19T22:07:41.853 INFO:teuthology.orchestra.run.smithi176.stderr:Waiting for mgr... 2024-02-19T22:07:41.853 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T22:07:42.831 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.829+0000 7fb9ab263700 1 Processor -- start 2024-02-19T22:07:42.832 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.829+0000 7fb9ab263700 1 -- start start 2024-02-19T22:07:42.832 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.829+0000 7fb9ab263700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a40703f0 0x7fb9a40fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:42.832 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.830+0000 7fb9ab263700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a40fef80 con 0x7fb9a40703f0 2024-02-19T22:07:42.832 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.830+0000 7fb9a8fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a40703f0 0x7fb9a40fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:42.832 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.830+0000 7fb9a8fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a40703f0 0x7fb9a40fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:55302/0 (socket says 172.21.15.176:55302) 2024-02-19T22:07:42.832 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.830+0000 7fb9a8fff700 1 -- 172.21.15.176:0/4145564300 learned_addr learned my addr 172.21.15.176:0/4145564300 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:42.833 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.830+0000 7fb9a8fff700 1 -- 172.21.15.176:0/4145564300 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a40ff0c0 con 0x7fb9a40703f0 2024-02-19T22:07:42.833 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.830+0000 7fb9a8fff700 1 --2- 172.21.15.176:0/4145564300 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a40703f0 0x7fb9a40fea90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fb99800b5f0 tx=0x7fb998006ee0).ready entity=mon.0 client_cookie=75b1a503eccd452 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:42.833 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.831+0000 7fb9a37fe700 1 -- 172.21.15.176:0/4145564300 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb99800d600 con 0x7fb9a40703f0 2024-02-19T22:07:42.833 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.831+0000 7fb9a37fe700 1 -- 172.21.15.176:0/4145564300 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb99800dbf0 con 0x7fb9a40703f0 2024-02-19T22:07:42.835 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.831+0000 7fb9a37fe700 1 -- 172.21.15.176:0/4145564300 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb998011ce0 con 0x7fb9a40703f0 2024-02-19T22:07:42.835 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.831+0000 7fb9ab263700 1 -- 172.21.15.176:0/4145564300 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a40703f0 msgr2=0x7fb9a40fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:42.835 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.831+0000 7fb9ab263700 1 --2- 172.21.15.176:0/4145564300 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a40703f0 0x7fb9a40fea90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fb99800b5f0 tx=0x7fb998006ee0).stop 2024-02-19T22:07:42.836 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.831+0000 7fb9ab263700 1 -- 172.21.15.176:0/4145564300 shutdown_connections 2024-02-19T22:07:42.836 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.831+0000 7fb9ab263700 1 --2- 172.21.15.176:0/4145564300 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a40703f0 0x7fb9a40fea90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:42.836 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.831+0000 7fb9ab263700 1 -- 172.21.15.176:0/4145564300 >> 172.21.15.176:0/4145564300 conn(0x7fb9a40f6530 msgr2=0x7fb9a40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:42.836 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.831+0000 7fb9ab263700 1 -- 172.21.15.176:0/4145564300 shutdown_connections 2024-02-19T22:07:42.836 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.831+0000 7fb9ab263700 1 -- 172.21.15.176:0/4145564300 wait complete. 2024-02-19T22:07:42.836 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.832+0000 7fb9ab263700 1 Processor -- start 2024-02-19T22:07:42.836 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.832+0000 7fb9ab263700 1 -- start start 2024-02-19T22:07:42.836 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.832+0000 7fb9ab263700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a40703f0 0x7fb9a40fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:42.836 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.832+0000 7fb9ab263700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a40fa990 con 0x7fb9a40703f0 2024-02-19T22:07:42.836 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.832+0000 7fb9a8fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a40703f0 0x7fb9a40fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:42.836 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.833+0000 7fb9a8fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a40703f0 0x7fb9a40fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:55306/0 (socket says 172.21.15.176:55306) 2024-02-19T22:07:42.837 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.833+0000 7fb9a8fff700 1 -- 172.21.15.176:0/2927825965 learned_addr learned my addr 172.21.15.176:0/2927825965 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:42.837 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.833+0000 7fb9a8fff700 1 -- 172.21.15.176:0/2927825965 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99800b200 con 0x7fb9a40703f0 2024-02-19T22:07:42.837 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.833+0000 7fb9a8fff700 1 --2- 172.21.15.176:0/2927825965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a40703f0 0x7fb9a40fc510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb9980115c0 tx=0x7fb998006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:42.837 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.833+0000 7fb9a17fa700 1 -- 172.21.15.176:0/2927825965 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb99801b410 con 0x7fb9a40703f0 2024-02-19T22:07:42.837 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.833+0000 7fb9a17fa700 1 -- 172.21.15.176:0/2927825965 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb99801ba00 con 0x7fb9a40703f0 2024-02-19T22:07:42.837 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.833+0000 7fb9a17fa700 1 -- 172.21.15.176:0/2927825965 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb99801aaa0 con 0x7fb9a40703f0 2024-02-19T22:07:42.837 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.833+0000 7fb9ab263700 1 -- 172.21.15.176:0/2927825965 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a40fab30 con 0x7fb9a40703f0 2024-02-19T22:07:42.837 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.833+0000 7fb9ab263700 1 -- 172.21.15.176:0/2927825965 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a40fafc0 con 0x7fb9a40703f0 2024-02-19T22:07:42.837 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.834+0000 7fb9a17fa700 1 -- 172.21.15.176:0/2927825965 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb99800f070 con 0x7fb9a40703f0 2024-02-19T22:07:42.837 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.834+0000 7fb9a17fa700 1 -- 172.21.15.176:0/2927825965 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb9980248c0 con 0x7fb9a40703f0 2024-02-19T22:07:42.837 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.834+0000 7fb9ab263700 1 -- 172.21.15.176:0/2927825965 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a4045ad0 con 0x7fb9a40703f0 2024-02-19T22:07:42.838 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.836+0000 7fb9a17fa700 1 -- 172.21.15.176:0/2927825965 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb998016070 con 0x7fb9a40703f0 2024-02-19T22:07:42.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.905+0000 7fb9ab263700 1 -- 172.21.15.176:0/2927825965 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb9a4104b00 con 0x7fb9a40703f0 2024-02-19T22:07:42.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.905+0000 7fb9a17fa700 1 -- 172.21.15.176:0/2927825965 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fb99801b570 con 0x7fb9a40703f0 2024-02-19T22:07:42.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: { 2024-02-19T22:07:42.913 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "fsid": "36a8f1ec-cf73-11ee-95bb-87774f69a715", 2024-02-19T22:07:42.913 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "health": { 2024-02-19T22:07:42.913 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T22:07:42.913 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T22:07:42.913 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T22:07:42.913 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:42.913 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T22:07:42.913 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T22:07:42.913 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 0 2024-02-19T22:07:42.913 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: ], 2024-02-19T22:07:42.913 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T22:07:42.913 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "smithi176" 2024-02-19T22:07:42.913 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: ], 2024-02-19T22:07:42.913 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "quorum_age": 3, 2024-02-19T22:07:42.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T22:07:42.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:07:42.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T22:07:42.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T22:07:42.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:42.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T22:07:42.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:07:42.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T22:07:42.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T22:07:42.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T22:07:42.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T22:07:42.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T22:07:42.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T22:07:42.915 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T22:07:42.916 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:42.916 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T22:07:42.916 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "available": false, 2024-02-19T22:07:42.916 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T22:07:42.916 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T22:07:42.916 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "iostat", 2024-02-19T22:07:42.916 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "restful" 2024-02-19T22:07:42.916 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: ], 2024-02-19T22:07:42.916 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:07:42.916 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:42.916 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T22:07:42.916 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:07:42.916 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "modified": "2024-02-19T22:07:34.781639+0000", 2024-02-19T22:07:42.916 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:07:42.917 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:42.917 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T22:07:42.917 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: } 2024-02-19T22:07:42.917 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.906+0000 7fb9ab263700 1 -- 172.21.15.176:0/2927825965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a40703f0 msgr2=0x7fb9a40fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:42.917 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.906+0000 7fb9ab263700 1 --2- 172.21.15.176:0/2927825965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a40703f0 0x7fb9a40fc510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fb9980115c0 tx=0x7fb998006ee0).stop 2024-02-19T22:07:42.917 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.906+0000 7fb9ab263700 1 -- 172.21.15.176:0/2927825965 shutdown_connections 2024-02-19T22:07:42.917 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.906+0000 7fb9ab263700 1 --2- 172.21.15.176:0/2927825965 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9a40703f0 0x7fb9a40fc510 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:42.917 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.906+0000 7fb9ab263700 1 -- 172.21.15.176:0/2927825965 >> 172.21.15.176:0/2927825965 conn(0x7fb9a40f6530 msgr2=0x7fb9a40f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:42.917 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.906+0000 7fb9ab263700 1 -- 172.21.15.176:0/2927825965 shutdown_connections 2024-02-19T22:07:42.917 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:42.906+0000 7fb9ab263700 1 -- 172.21.15.176:0/2927825965 wait complete. 2024-02-19T22:07:43.229 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:42 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:42.906997+0000 mon.smithi176 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.176:0/2927825965' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T22:07:43.527 INFO:teuthology.orchestra.run.smithi176.stderr:mgr not available, waiting (1/15)... 2024-02-19T22:07:45.529 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T22:07:46.041 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.040+0000 7f4753685700 1 Processor -- start 2024-02-19T22:07:46.041 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.040+0000 7f4753685700 1 -- start start 2024-02-19T22:07:46.041 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.041+0000 7f4753685700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f474c0703f0 0x7f474c100e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:46.042 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.041+0000 7f4753685700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c0fc880 con 0x7f474c0703f0 2024-02-19T22:07:46.042 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.041+0000 7f4751421700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f474c0703f0 0x7f474c100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:46.042 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.041+0000 7f4751421700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f474c0703f0 0x7f474c100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38440/0 (socket says 172.21.15.176:38440) 2024-02-19T22:07:46.042 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.041+0000 7f4751421700 1 -- 172.21.15.176:0/3796357470 learned_addr learned my addr 172.21.15.176:0/3796357470 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:46.042 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.041+0000 7f4751421700 1 -- 172.21.15.176:0/3796357470 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c101380 con 0x7f474c0703f0 2024-02-19T22:07:46.042 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.041+0000 7f4751421700 1 --2- 172.21.15.176:0/3796357470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f474c0703f0 0x7f474c100e90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f474800b5f0 tx=0x7f4748006ee0).ready entity=mon.0 client_cookie=c355a316c93900db server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:46.042 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.041+0000 7f473bfff700 1 -- 172.21.15.176:0/3796357470 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f474800d600 con 0x7f474c0703f0 2024-02-19T22:07:46.042 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.041+0000 7f473bfff700 1 -- 172.21.15.176:0/3796357470 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f474800dbf0 con 0x7f474c0703f0 2024-02-19T22:07:46.042 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.041+0000 7f473bfff700 1 -- 172.21.15.176:0/3796357470 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4748011ce0 con 0x7f474c0703f0 2024-02-19T22:07:46.044 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.042+0000 7f4753685700 1 -- 172.21.15.176:0/3796357470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f474c0703f0 msgr2=0x7f474c100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:46.044 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.042+0000 7f4753685700 1 --2- 172.21.15.176:0/3796357470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f474c0703f0 0x7f474c100e90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f474800b5f0 tx=0x7f4748006ee0).stop 2024-02-19T22:07:46.044 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.042+0000 7f4753685700 1 -- 172.21.15.176:0/3796357470 shutdown_connections 2024-02-19T22:07:46.045 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.042+0000 7f4753685700 1 --2- 172.21.15.176:0/3796357470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f474c0703f0 0x7f474c100e90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:46.045 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.042+0000 7f4753685700 1 -- 172.21.15.176:0/3796357470 >> 172.21.15.176:0/3796357470 conn(0x7f474c0f6530 msgr2=0x7f474c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:46.045 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.043+0000 7f4753685700 1 -- 172.21.15.176:0/3796357470 shutdown_connections 2024-02-19T22:07:46.045 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.043+0000 7f4753685700 1 -- 172.21.15.176:0/3796357470 wait complete. 2024-02-19T22:07:46.045 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.043+0000 7f4753685700 1 Processor -- start 2024-02-19T22:07:46.045 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.043+0000 7f4753685700 1 -- start start 2024-02-19T22:07:46.045 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.043+0000 7f4753685700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f474c0703f0 0x7f474c10ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:46.045 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.043+0000 7f4753685700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c10d2f0 con 0x7f474c0703f0 2024-02-19T22:07:46.045 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.043+0000 7f4751421700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f474c0703f0 0x7f474c10ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:46.045 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.043+0000 7f4751421700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f474c0703f0 0x7f474c10ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38444/0 (socket says 172.21.15.176:38444) 2024-02-19T22:07:46.045 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.043+0000 7f4751421700 1 -- 172.21.15.176:0/2142402573 learned_addr learned my addr 172.21.15.176:0/2142402573 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:46.045 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.044+0000 7f4751421700 1 -- 172.21.15.176:0/2142402573 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474800b200 con 0x7f474c0703f0 2024-02-19T22:07:46.045 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.044+0000 7f4751421700 1 --2- 172.21.15.176:0/2142402573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f474c0703f0 0x7f474c10ce00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f47480115c0 tx=0x7f4748006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:46.045 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.044+0000 7f4739ffb700 1 -- 172.21.15.176:0/2142402573 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f474801b410 con 0x7f474c0703f0 2024-02-19T22:07:46.046 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.044+0000 7f4739ffb700 1 -- 172.21.15.176:0/2142402573 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f474801ba00 con 0x7f474c0703f0 2024-02-19T22:07:46.046 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.044+0000 7f4739ffb700 1 -- 172.21.15.176:0/2142402573 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f474801aaa0 con 0x7f474c0703f0 2024-02-19T22:07:46.046 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.044+0000 7f4753685700 1 -- 172.21.15.176:0/2142402573 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474c10d4f0 con 0x7f474c0703f0 2024-02-19T22:07:46.046 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.044+0000 7f4753685700 1 -- 172.21.15.176:0/2142402573 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f474c10e100 con 0x7f474c0703f0 2024-02-19T22:07:46.046 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.045+0000 7f4739ffb700 1 -- 172.21.15.176:0/2142402573 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f474800f070 con 0x7f474c0703f0 2024-02-19T22:07:46.046 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.045+0000 7f4739ffb700 1 -- 172.21.15.176:0/2142402573 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4748024d20 con 0x7f474c0703f0 2024-02-19T22:07:46.046 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.045+0000 7f4753685700 1 -- 172.21.15.176:0/2142402573 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474c059100 con 0x7f474c0703f0 2024-02-19T22:07:46.047 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.047+0000 7f4739ffb700 1 -- 172.21.15.176:0/2142402573 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4748016070 con 0x7f474c0703f0 2024-02-19T22:07:46.122 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.116+0000 7f4753685700 1 -- 172.21.15.176:0/2142402573 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f474c1020a0 con 0x7f474c0703f0 2024-02-19T22:07:46.122 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.117+0000 7f4739ffb700 1 -- 172.21.15.176:0/2142402573 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f4748024410 con 0x7f474c0703f0 2024-02-19T22:07:46.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: { 2024-02-19T22:07:46.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "fsid": "36a8f1ec-cf73-11ee-95bb-87774f69a715", 2024-02-19T22:07:46.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "health": { 2024-02-19T22:07:46.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T22:07:46.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T22:07:46.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T22:07:46.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:46.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T22:07:46.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T22:07:46.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 0 2024-02-19T22:07:46.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: ], 2024-02-19T22:07:46.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T22:07:46.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "smithi176" 2024-02-19T22:07:46.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: ], 2024-02-19T22:07:46.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-02-19T22:07:46.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T22:07:46.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:07:46.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T22:07:46.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T22:07:46.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:46.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T22:07:46.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:07:46.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T22:07:46.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T22:07:46.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T22:07:46.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T22:07:46.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T22:07:46.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T22:07:46.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:46.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T22:07:46.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T22:07:46.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T22:07:46.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T22:07:46.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T22:07:46.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T22:07:46.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T22:07:46.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T22:07:46.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "available": false, 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "iostat", 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "restful" 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: ], 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:07:46.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:46.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T22:07:46.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:07:46.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "modified": "2024-02-19T22:07:34.781639+0000", 2024-02-19T22:07:46.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:07:46.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:46.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T22:07:46.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: } 2024-02-19T22:07:46.128 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.118+0000 7f4753685700 1 -- 172.21.15.176:0/2142402573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f474c0703f0 msgr2=0x7f474c10ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:46.128 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.118+0000 7f4753685700 1 --2- 172.21.15.176:0/2142402573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f474c0703f0 0x7f474c10ce00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f47480115c0 tx=0x7f4748006ee0).stop 2024-02-19T22:07:46.128 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.118+0000 7f4753685700 1 -- 172.21.15.176:0/2142402573 shutdown_connections 2024-02-19T22:07:46.128 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.118+0000 7f4753685700 1 --2- 172.21.15.176:0/2142402573 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f474c0703f0 0x7f474c10ce00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:46.128 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.118+0000 7f4753685700 1 -- 172.21.15.176:0/2142402573 >> 172.21.15.176:0/2142402573 conn(0x7f474c0f6530 msgr2=0x7f474c0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:46.128 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.119+0000 7f4753685700 1 -- 172.21.15.176:0/2142402573 shutdown_connections 2024-02-19T22:07:46.128 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:46.119+0000 7f4753685700 1 -- 172.21.15.176:0/2142402573 wait complete. 2024-02-19T22:07:46.401 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:46 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:46.117495+0000 mon.smithi176 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.176:0/2142402573' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T22:07:46.621 INFO:teuthology.orchestra.run.smithi176.stderr:mgr not available, waiting (2/15)... 2024-02-19T22:07:47.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:46.679553+0000 mon.smithi176 (mon.0) 9 : cluster [INF] Activating manager daemon smithi176.faoogw 2024-02-19T22:07:47.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:46.682664+0000 mon.smithi176 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi176.faoogw(active, starting, since 0.00319308s) 2024-02-19T22:07:47.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:46.683304+0000 mon.smithi176 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.176:0/4036596463' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:07:47.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:46.683440+0000 mon.smithi176 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.176:0/4036596463' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:07:47.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:46.683623+0000 mon.smithi176 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.176:0/4036596463' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:07:47.635 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:46.683810+0000 mon.smithi176 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.176:0/4036596463' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-02-19T22:07:47.635 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:46.684064+0000 mon.smithi176 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.176:0/4036596463' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mgr metadata", "who": "smithi176.faoogw", "id": "smithi176.faoogw"}]: dispatch 2024-02-19T22:07:47.635 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:46.685953+0000 mon.smithi176 (mon.0) 16 : cluster [INF] Manager daemon smithi176.faoogw is now available 2024-02-19T22:07:47.635 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:46.693043+0000 mon.smithi176 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.176:0/4036596463' entity='mgr.smithi176.faoogw' 2024-02-19T22:07:47.635 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:46.697412+0000 mon.smithi176 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.176:0/4036596463' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:07:47.635 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:46.698476+0000 mon.smithi176 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.176:0/4036596463' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:07:47.635 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:46.703232+0000 mon.smithi176 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.176:0/4036596463' entity='mgr.smithi176.faoogw' 2024-02-19T22:07:47.635 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:46.705087+0000 mon.smithi176 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.176:0/4036596463' entity='mgr.smithi176.faoogw' 2024-02-19T22:07:48.624 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T22:07:49.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:48 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:47.685552+0000 mon.smithi176 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi176.faoogw(active, since 1.00608s) 2024-02-19T22:07:49.906 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.905+0000 7ff42396a700 1 Processor -- start 2024-02-19T22:07:49.906 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.905+0000 7ff42396a700 1 -- start start 2024-02-19T22:07:49.906 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.905+0000 7ff42396a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41c0703b0 0x7ff41c100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:49.906 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.905+0000 7ff42396a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c0fc8c0 con 0x7ff41c0703b0 2024-02-19T22:07:49.906 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.906+0000 7ff421706700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41c0703b0 0x7ff41c100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:49.907 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.906+0000 7ff421706700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41c0703b0 0x7ff41c100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38476/0 (socket says 172.21.15.176:38476) 2024-02-19T22:07:49.907 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.906+0000 7ff421706700 1 -- 172.21.15.176:0/3170156334 learned_addr learned my addr 172.21.15.176:0/3170156334 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:49.907 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.906+0000 7ff421706700 1 -- 172.21.15.176:0/3170156334 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c1013c0 con 0x7ff41c0703b0 2024-02-19T22:07:49.907 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.906+0000 7ff421706700 1 --2- 172.21.15.176:0/3170156334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41c0703b0 0x7ff41c100ed0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff40c00b5f0 tx=0x7ff40c006ee0).ready entity=mon.0 client_cookie=d2340187dfd4716b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:49.907 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.906+0000 7ff40bfff700 1 -- 172.21.15.176:0/3170156334 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff40c00d600 con 0x7ff41c0703b0 2024-02-19T22:07:49.907 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.906+0000 7ff40bfff700 1 -- 172.21.15.176:0/3170156334 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff40c00dbf0 con 0x7ff41c0703b0 2024-02-19T22:07:49.907 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.906+0000 7ff40bfff700 1 -- 172.21.15.176:0/3170156334 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff40c011cb0 con 0x7ff41c0703b0 2024-02-19T22:07:49.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.907+0000 7ff42396a700 1 -- 172.21.15.176:0/3170156334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41c0703b0 msgr2=0x7ff41c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:49.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.907+0000 7ff42396a700 1 --2- 172.21.15.176:0/3170156334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41c0703b0 0x7ff41c100ed0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff40c00b5f0 tx=0x7ff40c006ee0).stop 2024-02-19T22:07:49.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.907+0000 7ff42396a700 1 -- 172.21.15.176:0/3170156334 shutdown_connections 2024-02-19T22:07:49.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.907+0000 7ff42396a700 1 --2- 172.21.15.176:0/3170156334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41c0703b0 0x7ff41c100ed0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:49.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.907+0000 7ff42396a700 1 -- 172.21.15.176:0/3170156334 >> 172.21.15.176:0/3170156334 conn(0x7ff41c0f6530 msgr2=0x7ff41c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:49.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.907+0000 7ff42396a700 1 -- 172.21.15.176:0/3170156334 shutdown_connections 2024-02-19T22:07:49.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.907+0000 7ff42396a700 1 -- 172.21.15.176:0/3170156334 wait complete. 2024-02-19T22:07:49.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.907+0000 7ff42396a700 1 Processor -- start 2024-02-19T22:07:49.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.908+0000 7ff42396a700 1 -- start start 2024-02-19T22:07:49.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.908+0000 7ff42396a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41c0703b0 0x7ff41c10cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:49.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.908+0000 7ff42396a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c10d270 con 0x7ff41c0703b0 2024-02-19T22:07:49.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.908+0000 7ff421706700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41c0703b0 0x7ff41c10cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:49.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.908+0000 7ff421706700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41c0703b0 0x7ff41c10cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38486/0 (socket says 172.21.15.176:38486) 2024-02-19T22:07:49.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.908+0000 7ff421706700 1 -- 172.21.15.176:0/3922626995 learned_addr learned my addr 172.21.15.176:0/3922626995 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:49.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.908+0000 7ff421706700 1 -- 172.21.15.176:0/3922626995 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40c00b200 con 0x7ff41c0703b0 2024-02-19T22:07:49.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.908+0000 7ff421706700 1 --2- 172.21.15.176:0/3922626995 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41c0703b0 0x7ff41c10cd80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7ff40c006800 tx=0x7ff40c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:49.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.908+0000 7ff409ffb700 1 -- 172.21.15.176:0/3922626995 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff40c01b570 con 0x7ff41c0703b0 2024-02-19T22:07:49.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.908+0000 7ff409ffb700 1 -- 172.21.15.176:0/3922626995 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff40c01bb60 con 0x7ff41c0703b0 2024-02-19T22:07:49.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.908+0000 7ff42396a700 1 -- 172.21.15.176:0/3922626995 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41c10d470 con 0x7ff41c0703b0 2024-02-19T22:07:49.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.908+0000 7ff409ffb700 1 -- 172.21.15.176:0/3922626995 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff40c01a9f0 con 0x7ff41c0703b0 2024-02-19T22:07:49.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.909+0000 7ff42396a700 1 -- 172.21.15.176:0/3922626995 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41c10e080 con 0x7ff41c0703b0 2024-02-19T22:07:49.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.910+0000 7ff409ffb700 1 -- 172.21.15.176:0/3922626995 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7ff40c00f070 con 0x7ff41c0703b0 2024-02-19T22:07:49.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.910+0000 7ff42396a700 1 -- 172.21.15.176:0/3922626995 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff41c059100 con 0x7ff41c0703b0 2024-02-19T22:07:49.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.910+0000 7ff409ffb700 1 --2- 172.21.15.176:0/3922626995 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ff40402fe10 0x7ff4040322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:49.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.910+0000 7ff409ffb700 1 -- 172.21.15.176:0/3922626995 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff40c041260 con 0x7ff41c0703b0 2024-02-19T22:07:49.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.910+0000 7ff420f05700 1 --2- 172.21.15.176:0/3922626995 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ff40402fe10 0x7ff4040322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:49.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.910+0000 7ff420f05700 1 --2- 172.21.15.176:0/3922626995 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ff40402fe10 0x7ff4040322d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7ff4180096a0 tx=0x7ff418006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:49.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:49.914+0000 7ff409ffb700 1 -- 172.21.15.176:0/3922626995 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff40c022b70 con 0x7ff41c0703b0 2024-02-19T22:07:50.122 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:50.116+0000 7ff42396a700 1 -- 172.21.15.176:0/3922626995 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff41c045ad0 con 0x7ff41c0703b0 2024-02-19T22:07:50.122 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:50.116+0000 7ff409ffb700 1 -- 172.21.15.176:0/3922626995 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7ff40c016070 con 0x7ff41c0703b0 2024-02-19T22:07:50.122 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:50.122 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: { 2024-02-19T22:07:50.122 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "fsid": "36a8f1ec-cf73-11ee-95bb-87774f69a715", 2024-02-19T22:07:50.122 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "health": { 2024-02-19T22:07:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T22:07:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T22:07:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T22:07:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T22:07:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T22:07:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 0 2024-02-19T22:07:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: ], 2024-02-19T22:07:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T22:07:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "smithi176" 2024-02-19T22:07:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: ], 2024-02-19T22:07:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-02-19T22:07:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T22:07:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:07:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T22:07:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T22:07:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T22:07:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:07:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T22:07:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T22:07:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T22:07:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T22:07:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T22:07:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T22:07:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:50.125 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T22:07:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "available": true, 2024-02-19T22:07:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T22:07:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T22:07:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "iostat", 2024-02-19T22:07:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "restful" 2024-02-19T22:07:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: ], 2024-02-19T22:07:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:07:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T22:07:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:07:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "modified": "2024-02-19T22:07:34.781639+0000", 2024-02-19T22:07:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:07:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: }, 2024-02-19T22:07:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T22:07:50.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: } 2024-02-19T22:07:50.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:50.118+0000 7ff42396a700 1 -- 172.21.15.176:0/3922626995 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ff40402fe10 msgr2=0x7ff4040322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:50.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:50.118+0000 7ff42396a700 1 --2- 172.21.15.176:0/3922626995 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ff40402fe10 0x7ff4040322d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7ff4180096a0 tx=0x7ff418006b40).stop 2024-02-19T22:07:50.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:50.118+0000 7ff42396a700 1 -- 172.21.15.176:0/3922626995 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41c0703b0 msgr2=0x7ff41c10cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:50.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:50.118+0000 7ff42396a700 1 --2- 172.21.15.176:0/3922626995 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41c0703b0 0x7ff41c10cd80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7ff40c006800 tx=0x7ff40c006ee0).stop 2024-02-19T22:07:50.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:50.118+0000 7ff42396a700 1 -- 172.21.15.176:0/3922626995 shutdown_connections 2024-02-19T22:07:50.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:50.118+0000 7ff42396a700 1 --2- 172.21.15.176:0/3922626995 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ff40402fe10 0x7ff4040322d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:50.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:50.118+0000 7ff42396a700 1 --2- 172.21.15.176:0/3922626995 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff41c0703b0 0x7ff41c10cd80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:50.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:50.118+0000 7ff42396a700 1 -- 172.21.15.176:0/3922626995 >> 172.21.15.176:0/3922626995 conn(0x7ff41c0f6530 msgr2=0x7ff41c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:50.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:50.118+0000 7ff42396a700 1 -- 172.21.15.176:0/3922626995 shutdown_connections 2024-02-19T22:07:50.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:50.118+0000 7ff42396a700 1 -- 172.21.15.176:0/3922626995 wait complete. 2024-02-19T22:07:50.674 INFO:teuthology.orchestra.run.smithi176.stderr:mgr is available 2024-02-19T22:07:50.675 INFO:teuthology.orchestra.run.smithi176.stderr:Enabling cephadm module... 2024-02-19T22:07:50.675 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-02-19T22:07:50.942 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:50 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:49.686266+0000 mon.smithi176 (mon.0) 23 : cluster [DBG] mgrmap e4: smithi176.faoogw(active, since 3s) 2024-02-19T22:07:50.942 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:50 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:50.116894+0000 mon.smithi176 (mon.0) 24 : audit [DBG] from='client.? 172.21.15.176:0/3922626995' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T22:07:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.179+0000 7f635a3a2700 1 Processor -- start 2024-02-19T22:07:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.179+0000 7f635a3a2700 1 -- start start 2024-02-19T22:07:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.179+0000 7f635a3a2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63540703b0 0x7f6354100d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.179+0000 7f635a3a2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63540fc700 con 0x7f63540703b0 2024-02-19T22:07:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.179+0000 7f63593a0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63540703b0 0x7f6354100d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.179+0000 7f63593a0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63540703b0 0x7f6354100d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38496/0 (socket says 172.21.15.176:38496) 2024-02-19T22:07:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.179+0000 7f63593a0700 1 -- 172.21.15.176:0/2005335911 learned_addr learned my addr 172.21.15.176:0/2005335911 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.179+0000 7f63593a0700 1 -- 172.21.15.176:0/2005335911 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6354101200 con 0x7f63540703b0 2024-02-19T22:07:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.179+0000 7f63593a0700 1 --2- 172.21.15.176:0/2005335911 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63540703b0 0x7f6354100d10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f634400b5f0 tx=0x7f6344006ee0).ready entity=mon.0 client_cookie=6ce7fffbb30f6927 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.180+0000 7f6343fff700 1 -- 172.21.15.176:0/2005335911 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f634400d600 con 0x7f63540703b0 2024-02-19T22:07:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.180+0000 7f6343fff700 1 -- 172.21.15.176:0/2005335911 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f634400dbf0 con 0x7f63540703b0 2024-02-19T22:07:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.180+0000 7f6343fff700 1 -- 172.21.15.176:0/2005335911 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6344011cb0 con 0x7f63540703b0 2024-02-19T22:07:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.180+0000 7f635a3a2700 1 -- 172.21.15.176:0/2005335911 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63540703b0 msgr2=0x7f6354100d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.180+0000 7f635a3a2700 1 --2- 172.21.15.176:0/2005335911 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63540703b0 0x7f6354100d10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f634400b5f0 tx=0x7f6344006ee0).stop 2024-02-19T22:07:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.180+0000 7f635a3a2700 1 -- 172.21.15.176:0/2005335911 shutdown_connections 2024-02-19T22:07:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.180+0000 7f635a3a2700 1 --2- 172.21.15.176:0/2005335911 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63540703b0 0x7f6354100d10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.180+0000 7f635a3a2700 1 -- 172.21.15.176:0/2005335911 >> 172.21.15.176:0/2005335911 conn(0x7f63540f6390 msgr2=0x7f63540f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.180+0000 7f635a3a2700 1 -- 172.21.15.176:0/2005335911 shutdown_connections 2024-02-19T22:07:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.180+0000 7f635a3a2700 1 -- 172.21.15.176:0/2005335911 wait complete. 2024-02-19T22:07:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.181+0000 7f635a3a2700 1 Processor -- start 2024-02-19T22:07:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.181+0000 7f635a3a2700 1 -- start start 2024-02-19T22:07:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.181+0000 7f635a3a2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63540703b0 0x7f635410cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.181+0000 7f635a3a2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635410d0b0 con 0x7f63540703b0 2024-02-19T22:07:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.181+0000 7f63593a0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63540703b0 0x7f635410cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.181+0000 7f63593a0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63540703b0 0x7f635410cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38500/0 (socket says 172.21.15.176:38500) 2024-02-19T22:07:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.181+0000 7f63593a0700 1 -- 172.21.15.176:0/1876062139 learned_addr learned my addr 172.21.15.176:0/1876062139 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.181+0000 7f63593a0700 1 -- 172.21.15.176:0/1876062139 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634400b200 con 0x7f63540703b0 2024-02-19T22:07:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.181+0000 7f63593a0700 1 --2- 172.21.15.176:0/1876062139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63540703b0 0x7f635410cbc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f634400df90 tx=0x7f6344006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.182+0000 7f6341ffb700 1 -- 172.21.15.176:0/1876062139 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f634401b410 con 0x7f63540703b0 2024-02-19T22:07:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.182+0000 7f635a3a2700 1 -- 172.21.15.176:0/1876062139 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635410d2b0 con 0x7f63540703b0 2024-02-19T22:07:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.182+0000 7f6341ffb700 1 -- 172.21.15.176:0/1876062139 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f634401ba00 con 0x7f63540703b0 2024-02-19T22:07:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.182+0000 7f6341ffb700 1 -- 172.21.15.176:0/1876062139 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f634401aa30 con 0x7f63540703b0 2024-02-19T22:07:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.182+0000 7f635a3a2700 1 -- 172.21.15.176:0/1876062139 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635410d740 con 0x7f63540703b0 2024-02-19T22:07:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.183+0000 7f6341ffb700 1 -- 172.21.15.176:0/1876062139 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f634400f070 con 0x7f63540703b0 2024-02-19T22:07:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.183+0000 7f6341ffb700 1 --2- 172.21.15.176:0/1876062139 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7f633c02fdc0 0x7f633c032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.183+0000 7f6341ffb700 1 -- 172.21.15.176:0/1876062139 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6344042070 con 0x7f63540703b0 2024-02-19T22:07:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.183+0000 7f6358b9f700 1 --2- 172.21.15.176:0/1876062139 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7f633c02fdc0 0x7f633c032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.183+0000 7f635a3a2700 1 -- 172.21.15.176:0/1876062139 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6354045ad0 con 0x7f63540703b0 2024-02-19T22:07:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.183+0000 7f6358b9f700 1 --2- 172.21.15.176:0/1876062139 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7f633c02fdc0 0x7f633c032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f63500096a0 tx=0x7f6350006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.187+0000 7f6341ffb700 1 -- 172.21.15.176:0/1876062139 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6344016070 con 0x7f63540703b0 2024-02-19T22:07:51.372 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.372+0000 7f635a3a2700 1 -- 172.21.15.176:0/1876062139 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f6354101e80 con 0x7f63540703b0 2024-02-19T22:07:51.687 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.687+0000 7f6341ffb700 1 -- 172.21.15.176:0/1876062139 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f6344025980 con 0x7f63540703b0 2024-02-19T22:07:51.688 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.688+0000 7f6341ffb700 1 -- 172.21.15.176:0/1876062139 <== mon.0 v2:172.21.15.176:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f6344022d20 con 0x7f63540703b0 2024-02-19T22:07:51.691 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.690+0000 7f635a3a2700 1 -- 172.21.15.176:0/1876062139 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7f633c02fdc0 msgr2=0x7f633c032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:51.691 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.690+0000 7f635a3a2700 1 --2- 172.21.15.176:0/1876062139 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7f633c02fdc0 0x7f633c032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f63500096a0 tx=0x7f6350006b40).stop 2024-02-19T22:07:51.691 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.690+0000 7f635a3a2700 1 -- 172.21.15.176:0/1876062139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63540703b0 msgr2=0x7f635410cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:51.691 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.690+0000 7f635a3a2700 1 --2- 172.21.15.176:0/1876062139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63540703b0 0x7f635410cbc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f634400df90 tx=0x7f6344006ee0).stop 2024-02-19T22:07:51.691 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.690+0000 7f635a3a2700 1 -- 172.21.15.176:0/1876062139 shutdown_connections 2024-02-19T22:07:51.691 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.690+0000 7f635a3a2700 1 --2- 172.21.15.176:0/1876062139 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7f633c02fdc0 0x7f633c032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:51.691 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.690+0000 7f635a3a2700 1 --2- 172.21.15.176:0/1876062139 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63540703b0 0x7f635410cbc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:51.692 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.690+0000 7f635a3a2700 1 -- 172.21.15.176:0/1876062139 >> 172.21.15.176:0/1876062139 conn(0x7f63540f6390 msgr2=0x7f63540f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:51.692 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.690+0000 7f635a3a2700 1 -- 172.21.15.176:0/1876062139 shutdown_connections 2024-02-19T22:07:51.692 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:51.690+0000 7f635a3a2700 1 -- 172.21.15.176:0/1876062139 wait complete. 2024-02-19T22:07:52.102 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:51 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:51.373155+0000 mon.smithi176 (mon.0) 25 : audit [INF] from='client.? 172.21.15.176:0/1876062139' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-19T22:07:52.347 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-19T22:07:52.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:52 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:51.689075+0000 mon.smithi176 (mon.0) 26 : audit [INF] from='client.? 172.21.15.176:0/1876062139' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-19T22:07:52.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:52 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:51.689135+0000 mon.smithi176 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi176.faoogw(active, since 5s) 2024-02-19T22:07:52.940 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.939+0000 7f15e974c700 1 Processor -- start 2024-02-19T22:07:52.940 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.939+0000 7f15e974c700 1 -- start start 2024-02-19T22:07:52.940 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.939+0000 7f15e974c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e40680e0 0x7f15e4068500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:52.941 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.939+0000 7f15e974c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e40697d0 con 0x7f15e40680e0 2024-02-19T22:07:52.941 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.940+0000 7f15e2ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e40680e0 0x7f15e4068500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:52.942 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.940+0000 7f15e2ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e40680e0 0x7f15e4068500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38532/0 (socket says 172.21.15.176:38532) 2024-02-19T22:07:52.942 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.940+0000 7f15e2ffd700 1 -- 172.21.15.176:0/1191439400 learned_addr learned my addr 172.21.15.176:0/1191439400 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:52.942 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.941+0000 7f15e2ffd700 1 -- 172.21.15.176:0/1191439400 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e40689f0 con 0x7f15e40680e0 2024-02-19T22:07:52.942 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.941+0000 7f15e2ffd700 1 --2- 172.21.15.176:0/1191439400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e40680e0 0x7f15e4068500 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f15d800b5f0 tx=0x7f15d8006ee0).ready entity=mon.0 client_cookie=aa16fa2e6cacd89a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:52.943 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.942+0000 7f15e1ffb700 1 -- 172.21.15.176:0/1191439400 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15d800d600 con 0x7f15e40680e0 2024-02-19T22:07:52.943 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.942+0000 7f15e1ffb700 1 -- 172.21.15.176:0/1191439400 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f15d800dbf0 con 0x7f15e40680e0 2024-02-19T22:07:52.944 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.942+0000 7f15e1ffb700 1 -- 172.21.15.176:0/1191439400 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15d8011cb0 con 0x7f15e40680e0 2024-02-19T22:07:52.944 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.942+0000 7f15e974c700 1 -- 172.21.15.176:0/1191439400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e40680e0 msgr2=0x7f15e4068500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:52.944 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.942+0000 7f15e974c700 1 --2- 172.21.15.176:0/1191439400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e40680e0 0x7f15e4068500 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f15d800b5f0 tx=0x7f15d8006ee0).stop 2024-02-19T22:07:52.944 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.943+0000 7f15e974c700 1 -- 172.21.15.176:0/1191439400 shutdown_connections 2024-02-19T22:07:52.944 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.943+0000 7f15e974c700 1 --2- 172.21.15.176:0/1191439400 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e40680e0 0x7f15e4068500 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0x7f15d800b5f0 tx=0x7f15d8006ee0).stop 2024-02-19T22:07:52.944 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.943+0000 7f15e974c700 1 -- 172.21.15.176:0/1191439400 >> 172.21.15.176:0/1191439400 conn(0x7f15e4063ba0 msgr2=0x7f15e4065fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:52.945 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.944+0000 7f15e974c700 1 -- 172.21.15.176:0/1191439400 shutdown_connections 2024-02-19T22:07:52.945 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.944+0000 7f15e974c700 1 -- 172.21.15.176:0/1191439400 wait complete. 2024-02-19T22:07:52.945 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.944+0000 7f15e974c700 1 Processor -- start 2024-02-19T22:07:52.946 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.945+0000 7f15e974c700 1 -- start start 2024-02-19T22:07:52.946 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.945+0000 7f15e974c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e4112980 0x7f15e4110e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:52.946 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.945+0000 7f15e974c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e4111330 con 0x7f15e4112980 2024-02-19T22:07:52.946 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.946+0000 7f15e2ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e4112980 0x7f15e4110e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:52.946 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.946+0000 7f15e2ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e4112980 0x7f15e4110e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38540/0 (socket says 172.21.15.176:38540) 2024-02-19T22:07:52.946 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.946+0000 7f15e2ffd700 1 -- 172.21.15.176:0/2908423166 learned_addr learned my addr 172.21.15.176:0/2908423166 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:52.947 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.947+0000 7f15e2ffd700 1 -- 172.21.15.176:0/2908423166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d800b200 con 0x7f15e4112980 2024-02-19T22:07:52.947 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.947+0000 7f15e2ffd700 1 --2- 172.21.15.176:0/2908423166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e4112980 0x7f15e4110e40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f15d800b5c0 tx=0x7f15d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:52.948 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.948+0000 7f15cb7fe700 1 -- 172.21.15.176:0/2908423166 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15d8011400 con 0x7f15e4112980 2024-02-19T22:07:52.949 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.948+0000 7f15cb7fe700 1 -- 172.21.15.176:0/2908423166 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f15d801b8a0 con 0x7f15e4112980 2024-02-19T22:07:52.949 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.948+0000 7f15cb7fe700 1 -- 172.21.15.176:0/2908423166 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15d801a8f0 con 0x7f15e4112980 2024-02-19T22:07:52.949 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.948+0000 7f15e974c700 1 -- 172.21.15.176:0/2908423166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e4111530 con 0x7f15e4112980 2024-02-19T22:07:52.949 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.948+0000 7f15e974c700 1 -- 172.21.15.176:0/2908423166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e41119c0 con 0x7f15e4112980 2024-02-19T22:07:52.951 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.950+0000 7f15cb7fe700 1 -- 172.21.15.176:0/2908423166 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f15d800f070 con 0x7f15e4112980 2024-02-19T22:07:52.951 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.950+0000 7f15cb7fe700 1 --2- 172.21.15.176:0/2908423166 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7f15d002ff10 0x7f15d00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:52.951 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.950+0000 7f15cb7fe700 1 -- 172.21.15.176:0/2908423166 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f15d80412f0 con 0x7f15e4112980 2024-02-19T22:07:52.951 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.950+0000 7f15e27fc700 1 -- 172.21.15.176:0/2908423166 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7f15d002ff10 msgr2=0x7f15d00323d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/910564779 2024-02-19T22:07:52.951 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.950+0000 7f15e27fc700 1 --2- 172.21.15.176:0/2908423166 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7f15d002ff10 0x7f15d00323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:07:52.951 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.950+0000 7f15e974c700 1 -- 172.21.15.176:0/2908423166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e4045ad0 con 0x7f15e4112980 2024-02-19T22:07:52.958 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:52.959+0000 7f15cb7fe700 1 -- 172.21.15.176:0/2908423166 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15d8016070 con 0x7f15e4112980 2024-02-19T22:07:53.151 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:53.151+0000 7f15e27fc700 1 -- 172.21.15.176:0/2908423166 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7f15d002ff10 msgr2=0x7f15d00323d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/910564779 2024-02-19T22:07:53.151 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:53.151+0000 7f15e27fc700 1 --2- 172.21.15.176:0/2908423166 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7f15d002ff10 0x7f15d00323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:07:53.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:53.158+0000 7f15e974c700 1 -- 172.21.15.176:0/2908423166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f15e4027a00 con 0x7f15e4112980 2024-02-19T22:07:53.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:53.159+0000 7f15cb7fe700 1 -- 172.21.15.176:0/2908423166 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f15d8019950 con 0x7f15e4112980 2024-02-19T22:07:53.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: { 2024-02-19T22:07:53.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-19T22:07:53.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "available": true, 2024-02-19T22:07:53.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "active_name": "smithi176.faoogw", 2024-02-19T22:07:53.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-19T22:07:53.160 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: } 2024-02-19T22:07:53.161 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:53.161+0000 7f15e974c700 1 -- 172.21.15.176:0/2908423166 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7f15d002ff10 msgr2=0x7f15d00323d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:07:53.161 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:53.161+0000 7f15e974c700 1 --2- 172.21.15.176:0/2908423166 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7f15d002ff10 0x7f15d00323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:07:53.161 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:53.161+0000 7f15e974c700 1 -- 172.21.15.176:0/2908423166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e4112980 msgr2=0x7f15e4110e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:53.162 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:53.161+0000 7f15e974c700 1 --2- 172.21.15.176:0/2908423166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e4112980 0x7f15e4110e40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f15d800b5c0 tx=0x7f15d8006ee0).stop 2024-02-19T22:07:53.162 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:53.161+0000 7f15e974c700 1 -- 172.21.15.176:0/2908423166 shutdown_connections 2024-02-19T22:07:53.162 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:53.161+0000 7f15e974c700 1 --2- 172.21.15.176:0/2908423166 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7f15d002ff10 0x7f15d00323d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:07:53.162 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:53.161+0000 7f15e974c700 1 --2- 172.21.15.176:0/2908423166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f15e4112980 0x7f15e4110e40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:53.162 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:53.161+0000 7f15e974c700 1 -- 172.21.15.176:0/2908423166 >> 172.21.15.176:0/2908423166 conn(0x7f15e4063ba0 msgr2=0x7f15e4066650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:53.162 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:53.161+0000 7f15e974c700 1 -- 172.21.15.176:0/2908423166 shutdown_connections 2024-02-19T22:07:53.162 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:53.162+0000 7f15e974c700 1 -- 172.21.15.176:0/2908423166 wait complete. 2024-02-19T22:07:53.778 INFO:teuthology.orchestra.run.smithi176.stderr:Waiting for the mgr to restart... 2024-02-19T22:07:53.778 INFO:teuthology.orchestra.run.smithi176.stderr:Waiting for mgr epoch 5... 2024-02-19T22:07:53.778 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-19T22:07:53.781 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:53.159614+0000 mon.smithi176 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.176:0/2908423166' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T22:07:55.893 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.205+0000 7ffb70a7c700 1 Processor -- start 2024-02-19T22:07:55.893 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.205+0000 7ffb70a7c700 1 -- start start 2024-02-19T22:07:55.893 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.205+0000 7ffb70a7c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb6c0fad90 0x7ffb6c0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:55.893 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.205+0000 7ffb70a7c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c0fb6a0 con 0x7ffb6c0fad90 2024-02-19T22:07:55.894 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.206+0000 7ffb6a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb6c0fad90 0x7ffb6c0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:55.894 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.206+0000 7ffb6a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb6c0fad90 0x7ffb6c0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:44802/0 (socket says 172.21.15.176:44802) 2024-02-19T22:07:55.894 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.206+0000 7ffb6a59c700 1 -- 172.21.15.176:0/3233755327 learned_addr learned my addr 172.21.15.176:0/3233755327 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:55.894 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.206+0000 7ffb6a59c700 1 -- 172.21.15.176:0/3233755327 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c0fb7e0 con 0x7ffb6c0fad90 2024-02-19T22:07:55.894 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.206+0000 7ffb6a59c700 1 --2- 172.21.15.176:0/3233755327 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb6c0fad90 0x7ffb6c0fb1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ffb5c00b5f0 tx=0x7ffb5c006ee0).ready entity=mon.0 client_cookie=d15d2737b00223c9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:55.894 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.206+0000 7ffb6959a700 1 -- 172.21.15.176:0/3233755327 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb5c00d600 con 0x7ffb6c0fad90 2024-02-19T22:07:55.894 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.206+0000 7ffb6959a700 1 -- 172.21.15.176:0/3233755327 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ffb5c00dbf0 con 0x7ffb6c0fad90 2024-02-19T22:07:55.894 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.206+0000 7ffb6959a700 1 -- 172.21.15.176:0/3233755327 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb5c011cb0 con 0x7ffb6c0fad90 2024-02-19T22:07:55.894 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.207+0000 7ffb70a7c700 1 -- 172.21.15.176:0/3233755327 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb6c0fad90 msgr2=0x7ffb6c0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:55.894 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.207+0000 7ffb70a7c700 1 --2- 172.21.15.176:0/3233755327 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb6c0fad90 0x7ffb6c0fb1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ffb5c00b5f0 tx=0x7ffb5c006ee0).stop 2024-02-19T22:07:55.894 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.207+0000 7ffb70a7c700 1 -- 172.21.15.176:0/3233755327 shutdown_connections 2024-02-19T22:07:55.894 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.207+0000 7ffb70a7c700 1 --2- 172.21.15.176:0/3233755327 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb6c0fad90 0x7ffb6c0fb1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:55.894 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.207+0000 7ffb70a7c700 1 -- 172.21.15.176:0/3233755327 >> 172.21.15.176:0/3233755327 conn(0x7ffb6c0f6530 msgr2=0x7ffb6c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:55.894 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.207+0000 7ffb70a7c700 1 -- 172.21.15.176:0/3233755327 shutdown_connections 2024-02-19T22:07:55.895 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.207+0000 7ffb70a7c700 1 -- 172.21.15.176:0/3233755327 wait complete. 2024-02-19T22:07:55.895 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.207+0000 7ffb70a7c700 1 Processor -- start 2024-02-19T22:07:55.895 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.207+0000 7ffb70a7c700 1 -- start start 2024-02-19T22:07:55.895 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.208+0000 7ffb70a7c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb6c0fad90 0x7ffb6c10d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:55.895 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.208+0000 7ffb70a7c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c070450 con 0x7ffb6c0fad90 2024-02-19T22:07:55.895 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.208+0000 7ffb6a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb6c0fad90 0x7ffb6c10d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:55.895 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.208+0000 7ffb6a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb6c0fad90 0x7ffb6c10d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:44812/0 (socket says 172.21.15.176:44812) 2024-02-19T22:07:55.895 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.208+0000 7ffb6a59c700 1 -- 172.21.15.176:0/1006102059 learned_addr learned my addr 172.21.15.176:0/1006102059 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:07:55.895 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.208+0000 7ffb6a59c700 1 -- 172.21.15.176:0/1006102059 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5c00b200 con 0x7ffb6c0fad90 2024-02-19T22:07:55.895 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.208+0000 7ffb6a59c700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb6c0fad90 0x7ffb6c10d860 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ffb5c011ea0 tx=0x7ffb5c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:55.895 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.208+0000 7ffb5affd700 1 -- 172.21.15.176:0/1006102059 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb5c01b570 con 0x7ffb6c0fad90 2024-02-19T22:07:55.895 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.208+0000 7ffb70a7c700 1 -- 172.21.15.176:0/1006102059 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6c070650 con 0x7ffb6c0fad90 2024-02-19T22:07:55.895 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.208+0000 7ffb5affd700 1 -- 172.21.15.176:0/1006102059 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ffb5c01bb60 con 0x7ffb6c0fad90 2024-02-19T22:07:55.896 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.208+0000 7ffb5affd700 1 -- 172.21.15.176:0/1006102059 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb5c01a8d0 con 0x7ffb6c0fad90 2024-02-19T22:07:55.896 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.208+0000 7ffb70a7c700 1 -- 172.21.15.176:0/1006102059 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6c10df10 con 0x7ffb6c0fad90 2024-02-19T22:07:55.896 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.209+0000 7ffb5affd700 1 -- 172.21.15.176:0/1006102059 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ffb5c00f070 con 0x7ffb6c0fad90 2024-02-19T22:07:55.896 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.209+0000 7ffb5affd700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ffb5002fe10 0x7ffb500322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:55.896 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.209+0000 7ffb5affd700 1 -- 172.21.15.176:0/1006102059 --> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb50032990 con 0x7ffb5002fe10 2024-02-19T22:07:55.896 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.209+0000 7ffb5affd700 1 -- 172.21.15.176:0/1006102059 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ffb5c041490 con 0x7ffb6c0fad90 2024-02-19T22:07:55.896 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.209+0000 7ffb69d9b700 1 -- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ffb5002fe10 msgr2=0x7ffb500322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/910564779 2024-02-19T22:07:55.896 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.209+0000 7ffb69d9b700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ffb5002fe10 0x7ffb500322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:07:55.896 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.410+0000 7ffb69d9b700 1 -- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ffb5002fe10 msgr2=0x7ffb500322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/910564779 2024-02-19T22:07:55.896 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.410+0000 7ffb69d9b700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ffb5002fe10 0x7ffb500322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:07:55.896 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.811+0000 7ffb69d9b700 1 -- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ffb5002fe10 msgr2=0x7ffb500322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/910564779 2024-02-19T22:07:55.896 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:55.811+0000 7ffb69d9b700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ffb5002fe10 0x7ffb500322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T22:07:56.612 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:56.612+0000 7ffb69d9b700 1 -- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ffb5002fe10 msgr2=0x7ffb500322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/910564779 2024-02-19T22:07:56.612 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:56.612+0000 7ffb69d9b700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ffb5002fe10 0x7ffb500322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T22:07:56.722 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:56.722+0000 7ffb5affd700 1 -- 172.21.15.176:0/1006102059 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7ffb5c022d80 con 0x7ffb6c0fad90 2024-02-19T22:07:56.722 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:56.722+0000 7ffb5affd700 1 -- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ffb5002fe10 msgr2=0x7ffb500322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:07:56.723 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:56.722+0000 7ffb5affd700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ffb5002fe10 0x7ffb500322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:07:57.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:56 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:56.669597+0000 mon.smithi176 (mon.0) 29 : cluster [INF] Active manager daemon smithi176.faoogw restarted 2024-02-19T22:07:57.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:56 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:56.669959+0000 mon.smithi176 (mon.0) 30 : cluster [INF] Activating manager daemon smithi176.faoogw 2024-02-19T22:07:57.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:56 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:56.671845+0000 mon.smithi176 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-19T22:07:57.728 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.728+0000 7ffb5affd700 1 -- 172.21.15.176:0/1006102059 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7ffb5c01b6d0 con 0x7ffb6c0fad90 2024-02-19T22:07:57.729 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.728+0000 7ffb5affd700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7ffb500338e0 0x7ffb50035cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:07:57.729 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.728+0000 7ffb5affd700 1 -- 172.21.15.176:0/1006102059 --> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb50032990 con 0x7ffb500338e0 2024-02-19T22:07:57.729 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.728+0000 7ffb69d9b700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7ffb500338e0 0x7ffb50035cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:07:57.729 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.729+0000 7ffb69d9b700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7ffb500338e0 0x7ffb50035cd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ffb600036d0 tx=0x7ffb60006ca0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:07:57.729 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.730+0000 7ffb5affd700 1 -- 172.21.15.176:0/1006102059 <== mgr.14114 v2:172.21.15.176:6800/2184395941 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7ffb50032990 con 0x7ffb500338e0 2024-02-19T22:07:57.736 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.736+0000 7ffb70a7c700 1 -- 172.21.15.176:0/1006102059 --> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ffb6c059100 con 0x7ffb500338e0 2024-02-19T22:07:57.736 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.736+0000 7ffb5affd700 1 -- 172.21.15.176:0/1006102059 <== mgr.14114 v2:172.21.15.176:6800/2184395941 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ffb6c059100 con 0x7ffb500338e0 2024-02-19T22:07:57.736 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: { 2024-02-19T22:07:57.738 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-19T22:07:57.738 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "initialized": true 2024-02-19T22:07:57.738 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: } 2024-02-19T22:07:57.738 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.736+0000 7ffb70a7c700 1 -- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7ffb500338e0 msgr2=0x7ffb50035cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:57.738 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.736+0000 7ffb70a7c700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7ffb500338e0 0x7ffb50035cd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ffb600036d0 tx=0x7ffb60006ca0).stop 2024-02-19T22:07:57.738 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.736+0000 7ffb70a7c700 1 -- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb6c0fad90 msgr2=0x7ffb6c10d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:07:57.738 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.736+0000 7ffb70a7c700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb6c0fad90 0x7ffb6c10d860 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ffb5c011ea0 tx=0x7ffb5c006ee0).stop 2024-02-19T22:07:57.738 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.736+0000 7ffb70a7c700 1 -- 172.21.15.176:0/1006102059 shutdown_connections 2024-02-19T22:07:57.738 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.736+0000 7ffb70a7c700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7ffb500338e0 0x7ffb50035cd0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:57.738 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.736+0000 7ffb70a7c700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffb6c0fad90 0x7ffb6c10d860 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:07:57.738 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.736+0000 7ffb70a7c700 1 --2- 172.21.15.176:0/1006102059 >> [v2:172.21.15.176:6800/910564779,v1:172.21.15.176:6801/910564779] conn(0x7ffb5002fe10 0x7ffb500322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:07:57.738 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.736+0000 7ffb70a7c700 1 -- 172.21.15.176:0/1006102059 >> 172.21.15.176:0/1006102059 conn(0x7ffb6c0f6530 msgr2=0x7ffb6c0f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:07:57.738 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.736+0000 7ffb70a7c700 1 -- 172.21.15.176:0/1006102059 shutdown_connections 2024-02-19T22:07:57.739 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:07:57.736+0000 7ffb70a7c700 1 -- 172.21.15.176:0/1006102059 wait complete. 2024-02-19T22:07:58.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:56.723327+0000 mon.smithi176 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi176.faoogw(active, starting, since 0.0534387s) 2024-02-19T22:07:58.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:56.725983+0000 mon.smithi176 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-02-19T22:07:58.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:56.726645+0000 mon.smithi176 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mgr metadata", "who": "smithi176.faoogw", "id": "smithi176.faoogw"}]: dispatch 2024-02-19T22:07:58.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:56.727329+0000 mon.smithi176 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:07:58.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:56.727466+0000 mon.smithi176 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:07:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:56.727599+0000 mon.smithi176 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:07:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:56.741032+0000 mon.smithi176 (mon.0) 38 : cluster [INF] Manager daemon smithi176.faoogw is now available 2024-02-19T22:07:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:56.754700+0000 mon.smithi176 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' 2024-02-19T22:07:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:56.757345+0000 mon.smithi176 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' 2024-02-19T22:07:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:56.759863+0000 mon.smithi176 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' 2024-02-19T22:07:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:56.848106+0000 mon.smithi176 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' 2024-02-19T22:07:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:56.848810+0000 mon.smithi176 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:07:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:56.850618+0000 mon.smithi176 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:07:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:56.852059+0000 mon.smithi176 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:07:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:56.861977+0000 mon.smithi176 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:07:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:56.863133+0000 mon.smithi176 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:07:59.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:58 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:57.730734+0000 mon.smithi176 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi176.faoogw(active, since 1.06084s) 2024-02-19T22:08:00.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:07:59 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:07:58.778709+0000 mon.smithi176 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi176.faoogw(active, since 2s) 2024-02-19T22:08:02.706 INFO:teuthology.orchestra.run.smithi176.stderr:mgr epoch 5 is available 2024-02-19T22:08:02.706 INFO:teuthology.orchestra.run.smithi176.stderr:Setting orchestrator backend to cephadm... 2024-02-19T22:08:02.707 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-02-19T22:08:03.246 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.244+0000 7f67970bd700 1 Processor -- start 2024-02-19T22:08:03.246 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.244+0000 7f67970bd700 1 -- start start 2024-02-19T22:08:03.246 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.244+0000 7f67970bd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67900fad90 0x7f67900fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:03.246 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.244+0000 7f67970bd700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67900fb6a0 con 0x7f67900fad90 2024-02-19T22:08:03.246 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.244+0000 7f6794e59700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67900fad90 0x7f67900fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:03.246 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.245+0000 7f6794e59700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67900fad90 0x7f67900fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:44876/0 (socket says 172.21.15.176:44876) 2024-02-19T22:08:03.247 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.245+0000 7f6794e59700 1 -- 172.21.15.176:0/771577251 learned_addr learned my addr 172.21.15.176:0/771577251 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:03.247 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.245+0000 7f6794e59700 1 -- 172.21.15.176:0/771577251 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67900fb7e0 con 0x7f67900fad90 2024-02-19T22:08:03.247 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.245+0000 7f6794e59700 1 --2- 172.21.15.176:0/771577251 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67900fad90 0x7f67900fb1b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f678400b5f0 tx=0x7f6784006ee0).ready entity=mon.0 client_cookie=da82aafb87478fd3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:03.247 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.245+0000 7f678f7fe700 1 -- 172.21.15.176:0/771577251 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f678400d600 con 0x7f67900fad90 2024-02-19T22:08:03.247 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.245+0000 7f678f7fe700 1 -- 172.21.15.176:0/771577251 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f678400dbf0 con 0x7f67900fad90 2024-02-19T22:08:03.247 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.245+0000 7f678f7fe700 1 -- 172.21.15.176:0/771577251 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6784011cb0 con 0x7f67900fad90 2024-02-19T22:08:03.247 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.246+0000 7f67970bd700 1 -- 172.21.15.176:0/771577251 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67900fad90 msgr2=0x7f67900fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:03.247 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.246+0000 7f67970bd700 1 --2- 172.21.15.176:0/771577251 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67900fad90 0x7f67900fb1b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f678400b5f0 tx=0x7f6784006ee0).stop 2024-02-19T22:08:03.247 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.246+0000 7f67970bd700 1 -- 172.21.15.176:0/771577251 shutdown_connections 2024-02-19T22:08:03.247 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.246+0000 7f67970bd700 1 --2- 172.21.15.176:0/771577251 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67900fad90 0x7f67900fb1b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:03.247 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.246+0000 7f67970bd700 1 -- 172.21.15.176:0/771577251 >> 172.21.15.176:0/771577251 conn(0x7f67900f6530 msgr2=0x7f67900f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:03.247 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.246+0000 7f67970bd700 1 -- 172.21.15.176:0/771577251 shutdown_connections 2024-02-19T22:08:03.247 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.246+0000 7f67970bd700 1 -- 172.21.15.176:0/771577251 wait complete. 2024-02-19T22:08:03.250 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.247+0000 7f67970bd700 1 Processor -- start 2024-02-19T22:08:03.250 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.247+0000 7f67970bd700 1 -- start start 2024-02-19T22:08:03.250 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.247+0000 7f67970bd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67900fad90 0x7f6790101040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:03.250 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.247+0000 7f67970bd700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67900ff4c0 con 0x7f67900fad90 2024-02-19T22:08:03.250 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.247+0000 7f6794e59700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67900fad90 0x7f6790101040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:03.250 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.247+0000 7f6794e59700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67900fad90 0x7f6790101040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:44886/0 (socket says 172.21.15.176:44886) 2024-02-19T22:08:03.250 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.247+0000 7f6794e59700 1 -- 172.21.15.176:0/2208803927 learned_addr learned my addr 172.21.15.176:0/2208803927 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:03.250 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.247+0000 7f6794e59700 1 -- 172.21.15.176:0/2208803927 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678400b200 con 0x7f67900fad90 2024-02-19T22:08:03.250 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.248+0000 7f6794e59700 1 --2- 172.21.15.176:0/2208803927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67900fad90 0x7f6790101040 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f6784002f40 tx=0x7f6784006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:03.250 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.248+0000 7f678d7fa700 1 -- 172.21.15.176:0/2208803927 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f678401b410 con 0x7f67900fad90 2024-02-19T22:08:03.250 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.248+0000 7f67970bd700 1 -- 172.21.15.176:0/2208803927 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67900ff660 con 0x7f67900fad90 2024-02-19T22:08:03.250 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.248+0000 7f678d7fa700 1 -- 172.21.15.176:0/2208803927 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f678401ba00 con 0x7f67900fad90 2024-02-19T22:08:03.251 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.248+0000 7f678d7fa700 1 -- 172.21.15.176:0/2208803927 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f678401a900 con 0x7f67900fad90 2024-02-19T22:08:03.251 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.248+0000 7f67970bd700 1 -- 172.21.15.176:0/2208803927 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67900ffaf0 con 0x7f67900fad90 2024-02-19T22:08:03.251 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.249+0000 7f678d7fa700 1 -- 172.21.15.176:0/2208803927 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f678400f070 con 0x7f67900fad90 2024-02-19T22:08:03.251 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.249+0000 7f678d7fa700 1 --2- 172.21.15.176:0/2208803927 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f677402fe10 0x7f67740322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:03.251 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.249+0000 7f678d7fa700 1 -- 172.21.15.176:0/2208803927 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f67840412f0 con 0x7f67900fad90 2024-02-19T22:08:03.251 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.249+0000 7f678ffff700 1 --2- 172.21.15.176:0/2208803927 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f677402fe10 0x7f67740322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:03.251 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.249+0000 7f67970bd700 1 -- 172.21.15.176:0/2208803927 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6790059100 con 0x7f67900fad90 2024-02-19T22:08:03.251 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.249+0000 7f678ffff700 1 --2- 172.21.15.176:0/2208803927 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f677402fe10 0x7f67740322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f677c0096a0 tx=0x7f677c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:03.253 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.254+0000 7f678d7fa700 1 -- 172.21.15.176:0/2208803927 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6784016070 con 0x7f67900fad90 2024-02-19T22:08:03.429 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.429+0000 7f67970bd700 1 -- 172.21.15.176:0/2208803927 --> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f67900fcb50 con 0x7f677402fe10 2024-02-19T22:08:03.433 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.434+0000 7f678d7fa700 1 -- 172.21.15.176:0/2208803927 <== mgr.14114 v2:172.21.15.176:6800/2184395941 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f67900fcb50 con 0x7f677402fe10 2024-02-19T22:08:03.436 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.436+0000 7f67970bd700 1 -- 172.21.15.176:0/2208803927 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f677402fe10 msgr2=0x7f67740322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:03.437 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.436+0000 7f67970bd700 1 --2- 172.21.15.176:0/2208803927 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f677402fe10 0x7f67740322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f677c0096a0 tx=0x7f677c006b40).stop 2024-02-19T22:08:03.437 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.436+0000 7f67970bd700 1 -- 172.21.15.176:0/2208803927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67900fad90 msgr2=0x7f6790101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:03.437 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.436+0000 7f67970bd700 1 --2- 172.21.15.176:0/2208803927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67900fad90 0x7f6790101040 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f6784002f40 tx=0x7f6784006ee0).stop 2024-02-19T22:08:03.437 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.436+0000 7f67970bd700 1 -- 172.21.15.176:0/2208803927 shutdown_connections 2024-02-19T22:08:03.437 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.436+0000 7f67970bd700 1 --2- 172.21.15.176:0/2208803927 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f677402fe10 0x7f67740322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:03.437 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.436+0000 7f67970bd700 1 --2- 172.21.15.176:0/2208803927 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f67900fad90 0x7f6790101040 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:03.437 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.436+0000 7f67970bd700 1 -- 172.21.15.176:0/2208803927 >> 172.21.15.176:0/2208803927 conn(0x7f67900f6530 msgr2=0x7f6790070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:03.437 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.437+0000 7f67970bd700 1 -- 172.21.15.176:0/2208803927 shutdown_connections 2024-02-19T22:08:03.437 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:03.437+0000 7f67970bd700 1 -- 172.21.15.176:0/2208803927 wait complete. 2024-02-19T22:08:04.036 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-02-19T22:08:04.532 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.530+0000 7f9821e30700 1 Processor -- start 2024-02-19T22:08:04.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.530+0000 7f9821e30700 1 -- start start 2024-02-19T22:08:04.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.530+0000 7f9821e30700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f981c0703f0 0x7f981c0fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:04.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.530+0000 7f9821e30700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c0fef80 con 0x7f981c0703f0 2024-02-19T22:08:04.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.531+0000 7f981b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f981c0703f0 0x7f981c0fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:04.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.531+0000 7f981b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f981c0703f0 0x7f981c0fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:51278/0 (socket says 172.21.15.176:51278) 2024-02-19T22:08:04.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.531+0000 7f981b7fe700 1 -- 172.21.15.176:0/3013063012 learned_addr learned my addr 172.21.15.176:0/3013063012 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:04.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.531+0000 7f981b7fe700 1 -- 172.21.15.176:0/3013063012 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c0ff0c0 con 0x7f981c0703f0 2024-02-19T22:08:04.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.531+0000 7f981b7fe700 1 --2- 172.21.15.176:0/3013063012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f981c0703f0 0x7f981c0fea90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f981000b5f0 tx=0x7f9810006ee0).ready entity=mon.0 client_cookie=a2136545c6708a07 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:04.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.531+0000 7f981a7fc700 1 -- 172.21.15.176:0/3013063012 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f981000d600 con 0x7f981c0703f0 2024-02-19T22:08:04.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.531+0000 7f981a7fc700 1 -- 172.21.15.176:0/3013063012 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f981000dbf0 con 0x7f981c0703f0 2024-02-19T22:08:04.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.531+0000 7f981a7fc700 1 -- 172.21.15.176:0/3013063012 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9810011ce0 con 0x7f981c0703f0 2024-02-19T22:08:04.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.532+0000 7f9821e30700 1 -- 172.21.15.176:0/3013063012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f981c0703f0 msgr2=0x7f981c0fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:04.534 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.532+0000 7f9821e30700 1 --2- 172.21.15.176:0/3013063012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f981c0703f0 0x7f981c0fea90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f981000b5f0 tx=0x7f9810006ee0).stop 2024-02-19T22:08:04.534 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.532+0000 7f9821e30700 1 -- 172.21.15.176:0/3013063012 shutdown_connections 2024-02-19T22:08:04.534 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.532+0000 7f9821e30700 1 --2- 172.21.15.176:0/3013063012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f981c0703f0 0x7f981c0fea90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:04.534 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.532+0000 7f9821e30700 1 -- 172.21.15.176:0/3013063012 >> 172.21.15.176:0/3013063012 conn(0x7f981c0f6530 msgr2=0x7f981c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:04.534 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.532+0000 7f9821e30700 1 -- 172.21.15.176:0/3013063012 shutdown_connections 2024-02-19T22:08:04.534 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.532+0000 7f9821e30700 1 -- 172.21.15.176:0/3013063012 wait complete. 2024-02-19T22:08:04.534 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.532+0000 7f9821e30700 1 Processor -- start 2024-02-19T22:08:04.534 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.533+0000 7f9821e30700 1 -- start start 2024-02-19T22:08:04.536 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.533+0000 7f9821e30700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f981c0703f0 0x7f981c10ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:04.536 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.533+0000 7f9821e30700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c10b040 con 0x7f981c0703f0 2024-02-19T22:08:04.536 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.533+0000 7f981b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f981c0703f0 0x7f981c10ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:04.536 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.533+0000 7f981b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f981c0703f0 0x7f981c10ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:51288/0 (socket says 172.21.15.176:51288) 2024-02-19T22:08:04.536 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.533+0000 7f981b7fe700 1 -- 172.21.15.176:0/1055124214 learned_addr learned my addr 172.21.15.176:0/1055124214 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:04.536 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.533+0000 7f981b7fe700 1 -- 172.21.15.176:0/1055124214 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981000b200 con 0x7f981c0703f0 2024-02-19T22:08:04.536 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.533+0000 7f981b7fe700 1 --2- 172.21.15.176:0/1055124214 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f981c0703f0 0x7f981c10ab50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f9810011ed0 tx=0x7f9810006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:04.536 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.534+0000 7f9803fff700 1 -- 172.21.15.176:0/1055124214 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f981001b570 con 0x7f981c0703f0 2024-02-19T22:08:04.536 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.534+0000 7f9803fff700 1 -- 172.21.15.176:0/1055124214 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f981001bb60 con 0x7f981c0703f0 2024-02-19T22:08:04.536 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.534+0000 7f9803fff700 1 -- 172.21.15.176:0/1055124214 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f981001aad0 con 0x7f981c0703f0 2024-02-19T22:08:04.537 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.534+0000 7f9821e30700 1 -- 172.21.15.176:0/1055124214 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981c10b240 con 0x7f981c0703f0 2024-02-19T22:08:04.537 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.534+0000 7f9821e30700 1 -- 172.21.15.176:0/1055124214 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c10b6d0 con 0x7f981c0703f0 2024-02-19T22:08:04.537 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.535+0000 7f9803fff700 1 -- 172.21.15.176:0/1055124214 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f981000f070 con 0x7f981c0703f0 2024-02-19T22:08:04.537 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.535+0000 7f9803fff700 1 --2- 172.21.15.176:0/1055124214 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f980802fe10 0x7f98080322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:04.537 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.535+0000 7f9803fff700 1 -- 172.21.15.176:0/1055124214 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f98100413d0 con 0x7f981c0703f0 2024-02-19T22:08:04.537 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.535+0000 7f9821e30700 1 -- 172.21.15.176:0/1055124214 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981c045ad0 con 0x7f981c0703f0 2024-02-19T22:08:04.537 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.535+0000 7f981affd700 1 --2- 172.21.15.176:0/1055124214 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f980802fe10 0x7f98080322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:04.537 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.535+0000 7f981affd700 1 --2- 172.21.15.176:0/1055124214 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f980802fe10 0x7f98080322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f980c0096a0 tx=0x7f980c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:04.539 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.539+0000 7f9803fff700 1 -- 172.21.15.176:0/1055124214 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9810016030 con 0x7f981c0703f0 2024-02-19T22:08:04.690 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.689+0000 7f9821e30700 1 -- 172.21.15.176:0/1055124214 --> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f981c10bf30 con 0x7f980802fe10 2024-02-19T22:08:04.690 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.690+0000 7f9803fff700 1 -- 172.21.15.176:0/1055124214 <== mgr.14114 v2:172.21.15.176:6800/2184395941 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f981c10bf30 con 0x7f980802fe10 2024-02-19T22:08:04.690 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: value unchanged 2024-02-19T22:08:04.692 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.691+0000 7f9821e30700 1 -- 172.21.15.176:0/1055124214 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f980802fe10 msgr2=0x7f98080322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:04.692 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.691+0000 7f9821e30700 1 --2- 172.21.15.176:0/1055124214 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f980802fe10 0x7f98080322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f980c0096a0 tx=0x7f980c006b40).stop 2024-02-19T22:08:04.692 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.691+0000 7f9821e30700 1 -- 172.21.15.176:0/1055124214 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f981c0703f0 msgr2=0x7f981c10ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:04.692 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.691+0000 7f9821e30700 1 --2- 172.21.15.176:0/1055124214 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f981c0703f0 0x7f981c10ab50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f9810011ed0 tx=0x7f9810006ee0).stop 2024-02-19T22:08:04.692 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.691+0000 7f9821e30700 1 -- 172.21.15.176:0/1055124214 shutdown_connections 2024-02-19T22:08:04.692 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.691+0000 7f9821e30700 1 --2- 172.21.15.176:0/1055124214 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f980802fe10 0x7f98080322d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:04.692 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.691+0000 7f9821e30700 1 --2- 172.21.15.176:0/1055124214 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f981c0703f0 0x7f981c10ab50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:04.692 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.691+0000 7f9821e30700 1 -- 172.21.15.176:0/1055124214 >> 172.21.15.176:0/1055124214 conn(0x7f981c0f6530 msgr2=0x7f981c0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:04.693 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.691+0000 7f9821e30700 1 -- 172.21.15.176:0/1055124214 shutdown_connections 2024-02-19T22:08:04.693 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:04.691+0000 7f9821e30700 1 -- 172.21.15.176:0/1055124214 wait complete. 2024-02-19T22:08:04.713 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:04 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:57.730109+0000 mgr.smithi176.faoogw (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T22:08:04.713 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:04 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:07:57.736754+0000 mgr.smithi176.faoogw (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T22:08:04.713 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:04 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:03.430383+0000 mgr.smithi176.faoogw (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:08:04.713 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:04 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:03.434340+0000 mon.smithi176 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:04.713 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:04 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:03.437470+0000 mon.smithi176 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:08:05.250 INFO:teuthology.orchestra.run.smithi176.stderr:Generating ssh key... 2024-02-19T22:08:05.250 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-02-19T22:08:05.817 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.815+0000 7fee00d58700 1 Processor -- start 2024-02-19T22:08:05.817 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.815+0000 7fee00d58700 1 -- start start 2024-02-19T22:08:05.818 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.815+0000 7fee00d58700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedfc0fad90 0x7fedfc0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:05.818 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.815+0000 7fee00d58700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc0fb6a0 con 0x7fedfc0fad90 2024-02-19T22:08:05.818 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.815+0000 7fedfa59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedfc0fad90 0x7fedfc0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:05.818 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.815+0000 7fedfa59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedfc0fad90 0x7fedfc0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:51294/0 (socket says 172.21.15.176:51294) 2024-02-19T22:08:05.818 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.815+0000 7fedfa59c700 1 -- 172.21.15.176:0/3590305759 learned_addr learned my addr 172.21.15.176:0/3590305759 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:05.818 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.815+0000 7fedfa59c700 1 -- 172.21.15.176:0/3590305759 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedfc0fb7e0 con 0x7fedfc0fad90 2024-02-19T22:08:05.818 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.815+0000 7fedfa59c700 1 --2- 172.21.15.176:0/3590305759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedfc0fad90 0x7fedfc0fb1b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fedec00b5f0 tx=0x7fedec006ee0).ready entity=mon.0 client_cookie=9408b536bf41ea4a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:05.818 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.816+0000 7fedf959a700 1 -- 172.21.15.176:0/3590305759 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedec00d600 con 0x7fedfc0fad90 2024-02-19T22:08:05.818 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.816+0000 7fedf959a700 1 -- 172.21.15.176:0/3590305759 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fedec00dbf0 con 0x7fedfc0fad90 2024-02-19T22:08:05.818 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.816+0000 7fedf959a700 1 -- 172.21.15.176:0/3590305759 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedec011cb0 con 0x7fedfc0fad90 2024-02-19T22:08:05.818 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.816+0000 7fee00d58700 1 -- 172.21.15.176:0/3590305759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedfc0fad90 msgr2=0x7fedfc0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:05.818 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.816+0000 7fee00d58700 1 --2- 172.21.15.176:0/3590305759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedfc0fad90 0x7fedfc0fb1b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fedec00b5f0 tx=0x7fedec006ee0).stop 2024-02-19T22:08:05.819 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.816+0000 7fee00d58700 1 -- 172.21.15.176:0/3590305759 shutdown_connections 2024-02-19T22:08:05.819 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.816+0000 7fee00d58700 1 --2- 172.21.15.176:0/3590305759 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedfc0fad90 0x7fedfc0fb1b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:05.819 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.816+0000 7fee00d58700 1 -- 172.21.15.176:0/3590305759 >> 172.21.15.176:0/3590305759 conn(0x7fedfc0f6530 msgr2=0x7fedfc0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:05.819 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.816+0000 7fee00d58700 1 -- 172.21.15.176:0/3590305759 shutdown_connections 2024-02-19T22:08:05.819 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.816+0000 7fee00d58700 1 -- 172.21.15.176:0/3590305759 wait complete. 2024-02-19T22:08:05.819 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.817+0000 7fee00d58700 1 Processor -- start 2024-02-19T22:08:05.819 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.817+0000 7fee00d58700 1 -- start start 2024-02-19T22:08:05.819 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.817+0000 7fee00d58700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedfc0fad90 0x7fedfc10d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:05.819 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.817+0000 7fee00d58700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc0704b0 con 0x7fedfc0fad90 2024-02-19T22:08:05.819 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.817+0000 7fedfa59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedfc0fad90 0x7fedfc10d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:05.819 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.817+0000 7fedfa59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedfc0fad90 0x7fedfc10d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:51308/0 (socket says 172.21.15.176:51308) 2024-02-19T22:08:05.819 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.817+0000 7fedfa59c700 1 -- 172.21.15.176:0/3261256861 learned_addr learned my addr 172.21.15.176:0/3261256861 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:05.819 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.818+0000 7fedfa59c700 1 -- 172.21.15.176:0/3261256861 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec00b200 con 0x7fedfc0fad90 2024-02-19T22:08:05.820 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.818+0000 7fedfa59c700 1 --2- 172.21.15.176:0/3261256861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedfc0fad90 0x7fedfc10d800 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fedec011ea0 tx=0x7fedec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:05.820 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.818+0000 7fedeaffd700 1 -- 172.21.15.176:0/3261256861 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedec01b570 con 0x7fedfc0fad90 2024-02-19T22:08:05.821 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.818+0000 7fedeaffd700 1 -- 172.21.15.176:0/3261256861 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fedec01bb60 con 0x7fedfc0fad90 2024-02-19T22:08:05.821 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.818+0000 7fee00d58700 1 -- 172.21.15.176:0/3261256861 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedfc0706b0 con 0x7fedfc0fad90 2024-02-19T22:08:05.821 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.818+0000 7fedeaffd700 1 -- 172.21.15.176:0/3261256861 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedec01a8d0 con 0x7fedfc0fad90 2024-02-19T22:08:05.821 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.818+0000 7fee00d58700 1 -- 172.21.15.176:0/3261256861 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedfc10df30 con 0x7fedfc0fad90 2024-02-19T22:08:05.821 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.819+0000 7fedeaffd700 1 -- 172.21.15.176:0/3261256861 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fedec00f070 con 0x7fedfc0fad90 2024-02-19T22:08:05.821 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.819+0000 7fee00d58700 1 -- 172.21.15.176:0/3261256861 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedfc045ad0 con 0x7fedfc0fad90 2024-02-19T22:08:05.821 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.819+0000 7fedeaffd700 1 --2- 172.21.15.176:0/3261256861 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7fede002fe10 0x7fede00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:05.821 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.819+0000 7fedeaffd700 1 -- 172.21.15.176:0/3261256861 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fedec041450 con 0x7fedfc0fad90 2024-02-19T22:08:05.821 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.819+0000 7fedf9d9b700 1 --2- 172.21.15.176:0/3261256861 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7fede002fe10 0x7fede00322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:05.821 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.819+0000 7fedf9d9b700 1 --2- 172.21.15.176:0/3261256861 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7fede002fe10 0x7fede00322d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fedf00096a0 tx=0x7fedf0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:05.822 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.823+0000 7fedeaffd700 1 -- 172.21.15.176:0/3261256861 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fedec022b70 con 0x7fedfc0fad90 2024-02-19T22:08:05.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:04.690498+0000 mgr.smithi176.faoogw (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:08:05.973 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:05.973+0000 7fee00d58700 1 -- 172.21.15.176:0/3261256861 --> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fedfc057f10 con 0x7fede002fe10 2024-02-19T22:08:06.156 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:06.156+0000 7fedeaffd700 1 -- 172.21.15.176:0/3261256861 <== mgr.14114 v2:172.21.15.176:6800/2184395941 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fedfc057f10 con 0x7fede002fe10 2024-02-19T22:08:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:06.158+0000 7fee00d58700 1 -- 172.21.15.176:0/3261256861 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7fede002fe10 msgr2=0x7fede00322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:06.158+0000 7fee00d58700 1 --2- 172.21.15.176:0/3261256861 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7fede002fe10 0x7fede00322d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fedf00096a0 tx=0x7fedf0006b40).stop 2024-02-19T22:08:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:06.158+0000 7fee00d58700 1 -- 172.21.15.176:0/3261256861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedfc0fad90 msgr2=0x7fedfc10d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:06.158+0000 7fee00d58700 1 --2- 172.21.15.176:0/3261256861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedfc0fad90 0x7fedfc10d800 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fedec011ea0 tx=0x7fedec006ee0).stop 2024-02-19T22:08:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:06.158+0000 7fee00d58700 1 -- 172.21.15.176:0/3261256861 shutdown_connections 2024-02-19T22:08:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:06.158+0000 7fee00d58700 1 --2- 172.21.15.176:0/3261256861 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7fede002fe10 0x7fede00322d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:06.158+0000 7fee00d58700 1 --2- 172.21.15.176:0/3261256861 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fedfc0fad90 0x7fedfc10d800 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:06.158+0000 7fee00d58700 1 -- 172.21.15.176:0/3261256861 >> 172.21.15.176:0/3261256861 conn(0x7fedfc0f6530 msgr2=0x7fedfc0f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:06.158+0000 7fee00d58700 1 -- 172.21.15.176:0/3261256861 shutdown_connections 2024-02-19T22:08:06.159 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:06.158+0000 7fee00d58700 1 -- 172.21.15.176:0/3261256861 wait complete. 2024-02-19T22:08:06.757 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-02-19T22:08:07.264 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.236+0000 7f599f79d700 1 Processor -- start 2024-02-19T22:08:07.265 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.237+0000 7f599f79d700 1 -- start start 2024-02-19T22:08:07.265 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.237+0000 7f599f79d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59980fbea0 0x7f59980fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:07.265 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.237+0000 7f599f79d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59980fc7b0 con 0x7f59980fbea0 2024-02-19T22:08:07.265 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.237+0000 7f599d539700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59980fbea0 0x7f59980fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:07.265 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.237+0000 7f599d539700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59980fbea0 0x7f59980fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:51312/0 (socket says 172.21.15.176:51312) 2024-02-19T22:08:07.265 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.237+0000 7f599d539700 1 -- 172.21.15.176:0/3467221307 learned_addr learned my addr 172.21.15.176:0/3467221307 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:07.265 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.237+0000 7f599d539700 1 -- 172.21.15.176:0/3467221307 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59980fc8f0 con 0x7f59980fbea0 2024-02-19T22:08:07.265 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.237+0000 7f599d539700 1 --2- 172.21.15.176:0/3467221307 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59980fbea0 0x7f59980fc2c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f598800b5f0 tx=0x7f5988006ee0).ready entity=mon.0 client_cookie=1a9f3d74ae7affc8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:07.265 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.237+0000 7f5987fff700 1 -- 172.21.15.176:0/3467221307 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f598800d600 con 0x7f59980fbea0 2024-02-19T22:08:07.265 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.237+0000 7f5987fff700 1 -- 172.21.15.176:0/3467221307 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f598800dbf0 con 0x7f59980fbea0 2024-02-19T22:08:07.265 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.237+0000 7f5987fff700 1 -- 172.21.15.176:0/3467221307 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5988011cb0 con 0x7f59980fbea0 2024-02-19T22:08:07.268 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.238+0000 7f599f79d700 1 -- 172.21.15.176:0/3467221307 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59980fbea0 msgr2=0x7f59980fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:07.268 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.238+0000 7f599f79d700 1 --2- 172.21.15.176:0/3467221307 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59980fbea0 0x7f59980fc2c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f598800b5f0 tx=0x7f5988006ee0).stop 2024-02-19T22:08:07.269 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.238+0000 7f599f79d700 1 -- 172.21.15.176:0/3467221307 shutdown_connections 2024-02-19T22:08:07.269 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.238+0000 7f599f79d700 1 --2- 172.21.15.176:0/3467221307 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59980fbea0 0x7f59980fc2c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:07.269 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.238+0000 7f599f79d700 1 -- 172.21.15.176:0/3467221307 >> 172.21.15.176:0/3467221307 conn(0x7f59980f76a0 msgr2=0x7f59980f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:07.269 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.238+0000 7f599f79d700 1 -- 172.21.15.176:0/3467221307 shutdown_connections 2024-02-19T22:08:07.269 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.238+0000 7f599f79d700 1 -- 172.21.15.176:0/3467221307 wait complete. 2024-02-19T22:08:07.269 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.238+0000 7f599f79d700 1 Processor -- start 2024-02-19T22:08:07.269 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.238+0000 7f599f79d700 1 -- start start 2024-02-19T22:08:07.269 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.239+0000 7f599f79d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59980fbea0 0x7f599810ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:07.269 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.239+0000 7f599f79d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599810b080 con 0x7f59980fbea0 2024-02-19T22:08:07.269 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.239+0000 7f599d539700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59980fbea0 0x7f599810ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:07.269 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.239+0000 7f599d539700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59980fbea0 0x7f599810ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:51328/0 (socket says 172.21.15.176:51328) 2024-02-19T22:08:07.269 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.239+0000 7f599d539700 1 -- 172.21.15.176:0/3811349534 learned_addr learned my addr 172.21.15.176:0/3811349534 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:07.269 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.239+0000 7f599d539700 1 -- 172.21.15.176:0/3811349534 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598800b200 con 0x7f59980fbea0 2024-02-19T22:08:07.269 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.239+0000 7f599d539700 1 --2- 172.21.15.176:0/3811349534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59980fbea0 0x7f599810ab90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f598800bf40 tx=0x7f5988006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:07.270 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.239+0000 7f5985ffb700 1 -- 172.21.15.176:0/3811349534 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f598801b570 con 0x7f59980fbea0 2024-02-19T22:08:07.270 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.239+0000 7f5985ffb700 1 -- 172.21.15.176:0/3811349534 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f598801bb60 con 0x7f59980fbea0 2024-02-19T22:08:07.270 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.239+0000 7f599f79d700 1 -- 172.21.15.176:0/3811349534 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599810b280 con 0x7f59980fbea0 2024-02-19T22:08:07.270 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.239+0000 7f5985ffb700 1 -- 172.21.15.176:0/3811349534 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f598801aa20 con 0x7f59980fbea0 2024-02-19T22:08:07.270 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.239+0000 7f599f79d700 1 -- 172.21.15.176:0/3811349534 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f599810b710 con 0x7f59980fbea0 2024-02-19T22:08:07.270 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.240+0000 7f5985ffb700 1 -- 172.21.15.176:0/3811349534 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f598800f070 con 0x7f59980fbea0 2024-02-19T22:08:07.270 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.240+0000 7f599f79d700 1 -- 172.21.15.176:0/3811349534 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5998045ad0 con 0x7f59980fbea0 2024-02-19T22:08:07.270 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.240+0000 7f5985ffb700 1 --2- 172.21.15.176:0/3811349534 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f598002fe70 0x7f5980032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:07.270 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.240+0000 7f5985ffb700 1 -- 172.21.15.176:0/3811349534 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5988041420 con 0x7f59980fbea0 2024-02-19T22:08:07.270 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.240+0000 7f599cd38700 1 --2- 172.21.15.176:0/3811349534 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f598002fe70 0x7f5980032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:07.270 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.241+0000 7f599cd38700 1 --2- 172.21.15.176:0/3811349534 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f598002fe70 0x7f5980032330 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f59940096a0 tx=0x7f5994006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:07.270 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.244+0000 7f5985ffb700 1 -- 172.21.15.176:0/3811349534 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5988019af0 con 0x7f59980fbea0 2024-02-19T22:08:07.398 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.398+0000 7f599f79d700 1 -- 172.21.15.176:0/3811349534 --> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f599810c030 con 0x7f598002fe70 2024-02-19T22:08:07.399 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.399+0000 7f5985ffb700 1 -- 172.21.15.176:0/3811349534 <== mgr.14114 v2:172.21.15.176:6800/2184395941 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f599810c030 con 0x7f598002fe70 2024-02-19T22:08:07.399 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwHsNZYEucPRxwPXghnjq8kZ2PKOCEiilJi9d8hHw/TyriGZ679M7DS5IuaOIIuCtBTY+dfMO8G5Yp4qIdpRMq0/1TZbINCvaVNmk7DgRmF0XcwfkbxHlo5Lw6NHG1/GyUv/8yJ1T8zmsyXbn5KXHpMINSnigvhjJbvX6Gzz8Lyr5aq/XOFBsBlFJzYKQHKmyCo1rhvBnlxcpB3NZ3wRGNqJetp0JsbQ36YFOGHjPlnLnvkcGOS7JD53jUVvxuoGMtvNwGhaunh028xgvCXMAV+FC8KuntEmJsATnbAZ7VwNWS2rDm3Wa6eMGGvmquQmMRuIh9+1UTZHWG7ylpNsqbOOTtfKNBOYMmvU6vj7RhCHh9VjkCJNz1AbNINv/ArfJt2/uRQ2HBJ8kZNWl6N1WFCtJ+tbjvQv3hggHxs3gndcchyJXpQ7GO2tok/3ePxwRKyCLtajHtT6OvdICJuF0Xye7P/xEFXMcpuLuiKyYJCMVkABjJI2lGtF553vbPtwc= ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:08:07.401 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.401+0000 7f599f79d700 1 -- 172.21.15.176:0/3811349534 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f598002fe70 msgr2=0x7f5980032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:07.401 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.401+0000 7f599f79d700 1 --2- 172.21.15.176:0/3811349534 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f598002fe70 0x7f5980032330 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f59940096a0 tx=0x7f5994006b40).stop 2024-02-19T22:08:07.401 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.401+0000 7f599f79d700 1 -- 172.21.15.176:0/3811349534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59980fbea0 msgr2=0x7f599810ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:07.401 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.401+0000 7f599f79d700 1 --2- 172.21.15.176:0/3811349534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59980fbea0 0x7f599810ab90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f598800bf40 tx=0x7f5988006ee0).stop 2024-02-19T22:08:07.401 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.401+0000 7f599f79d700 1 -- 172.21.15.176:0/3811349534 shutdown_connections 2024-02-19T22:08:07.402 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.401+0000 7f599f79d700 1 --2- 172.21.15.176:0/3811349534 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f598002fe70 0x7f5980032330 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:07.402 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.401+0000 7f599f79d700 1 --2- 172.21.15.176:0/3811349534 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59980fbea0 0x7f599810ab90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:07.402 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.401+0000 7f599f79d700 1 -- 172.21.15.176:0/3811349534 >> 172.21.15.176:0/3811349534 conn(0x7f59980f76a0 msgr2=0x7f59980f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:07.402 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.401+0000 7f599f79d700 1 -- 172.21.15.176:0/3811349534 shutdown_connections 2024-02-19T22:08:07.402 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:07.401+0000 7f599f79d700 1 -- 172.21.15.176:0/3811349534 wait complete. 2024-02-19T22:08:07.423 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:08:05.974542+0000 mgr.smithi176.faoogw (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-02-19T22:08:07.423 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:06.154679+0000 mon.smithi176 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:07.423 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:06.156430+0000 mon.smithi176 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:08.105 INFO:teuthology.orchestra.run.smithi176.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:08:08.106 INFO:teuthology.orchestra.run.smithi176.stderr:Adding key to root@localhost authorized_keys... 2024-02-19T22:08:08.106 INFO:teuthology.orchestra.run.smithi176.stderr:Adding host smithi176... 2024-02-19T22:08:08.106 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi176 172.21.15.176 2024-02-19T22:08:08.359 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:08 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:07.399869+0000 mgr.smithi176.faoogw (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:08:08.607 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.606+0000 7f014a410700 1 Processor -- start 2024-02-19T22:08:08.608 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.606+0000 7f014a410700 1 -- start start 2024-02-19T22:08:08.608 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.606+0000 7f014a410700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01440703f0 0x7f01440fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:08.608 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.606+0000 7f014a410700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01440fef80 con 0x7f01440703f0 2024-02-19T22:08:08.608 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.606+0000 7f0143fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01440703f0 0x7f01440fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:08.608 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.607+0000 7f0143fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01440703f0 0x7f01440fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:51342/0 (socket says 172.21.15.176:51342) 2024-02-19T22:08:08.608 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.607+0000 7f0143fff700 1 -- 172.21.15.176:0/44573171 learned_addr learned my addr 172.21.15.176:0/44573171 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:08.608 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.607+0000 7f0143fff700 1 -- 172.21.15.176:0/44573171 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01440ff0c0 con 0x7f01440703f0 2024-02-19T22:08:08.608 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.607+0000 7f0143fff700 1 --2- 172.21.15.176:0/44573171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01440703f0 0x7f01440fea90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f013400b5f0 tx=0x7f0134006ee0).ready entity=mon.0 client_cookie=22ee601a1415bca5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:08.608 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.607+0000 7f0142ffd700 1 -- 172.21.15.176:0/44573171 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f013400d600 con 0x7f01440703f0 2024-02-19T22:08:08.608 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.607+0000 7f0142ffd700 1 -- 172.21.15.176:0/44573171 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f013400dbf0 con 0x7f01440703f0 2024-02-19T22:08:08.608 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.607+0000 7f0142ffd700 1 -- 172.21.15.176:0/44573171 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0134011ce0 con 0x7f01440703f0 2024-02-19T22:08:08.608 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.608+0000 7f014a410700 1 -- 172.21.15.176:0/44573171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01440703f0 msgr2=0x7f01440fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:08.608 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.608+0000 7f014a410700 1 --2- 172.21.15.176:0/44573171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01440703f0 0x7f01440fea90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f013400b5f0 tx=0x7f0134006ee0).stop 2024-02-19T22:08:08.610 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.608+0000 7f014a410700 1 -- 172.21.15.176:0/44573171 shutdown_connections 2024-02-19T22:08:08.610 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.608+0000 7f014a410700 1 --2- 172.21.15.176:0/44573171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01440703f0 0x7f01440fea90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:08.610 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.608+0000 7f014a410700 1 -- 172.21.15.176:0/44573171 >> 172.21.15.176:0/44573171 conn(0x7f01440f6530 msgr2=0x7f01440f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:08.610 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.608+0000 7f014a410700 1 -- 172.21.15.176:0/44573171 shutdown_connections 2024-02-19T22:08:08.610 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.608+0000 7f014a410700 1 -- 172.21.15.176:0/44573171 wait complete. 2024-02-19T22:08:08.610 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.608+0000 7f014a410700 1 Processor -- start 2024-02-19T22:08:08.610 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.608+0000 7f014a410700 1 -- start start 2024-02-19T22:08:08.610 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.609+0000 7f014a410700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01440703f0 0x7f01440fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:08.610 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.609+0000 7f014a410700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01440fa990 con 0x7f01440703f0 2024-02-19T22:08:08.611 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.609+0000 7f0143fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01440703f0 0x7f01440fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:08.611 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.609+0000 7f0143fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01440703f0 0x7f01440fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:51350/0 (socket says 172.21.15.176:51350) 2024-02-19T22:08:08.611 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.609+0000 7f0143fff700 1 -- 172.21.15.176:0/3604886513 learned_addr learned my addr 172.21.15.176:0/3604886513 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:08.611 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.609+0000 7f0143fff700 1 -- 172.21.15.176:0/3604886513 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013400b200 con 0x7f01440703f0 2024-02-19T22:08:08.611 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.609+0000 7f0143fff700 1 --2- 172.21.15.176:0/3604886513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01440703f0 0x7f01440fc510 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f0134011ed0 tx=0x7f0134006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:08.611 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.609+0000 7f0140ff9700 1 -- 172.21.15.176:0/3604886513 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f013401b570 con 0x7f01440703f0 2024-02-19T22:08:08.611 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.609+0000 7f014a410700 1 -- 172.21.15.176:0/3604886513 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01440fab30 con 0x7f01440703f0 2024-02-19T22:08:08.611 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.609+0000 7f0140ff9700 1 -- 172.21.15.176:0/3604886513 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f013401bb60 con 0x7f01440703f0 2024-02-19T22:08:08.611 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.609+0000 7f0140ff9700 1 -- 172.21.15.176:0/3604886513 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f013401aad0 con 0x7f01440703f0 2024-02-19T22:08:08.611 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.609+0000 7f014a410700 1 -- 172.21.15.176:0/3604886513 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01440fafc0 con 0x7f01440703f0 2024-02-19T22:08:08.611 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.610+0000 7f0140ff9700 1 -- 172.21.15.176:0/3604886513 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f013400f070 con 0x7f01440703f0 2024-02-19T22:08:08.611 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.610+0000 7f0140ff9700 1 --2- 172.21.15.176:0/3604886513 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f013002fe10 0x7f01300322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:08.612 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.610+0000 7f0140ff9700 1 -- 172.21.15.176:0/3604886513 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0134041350 con 0x7f01440703f0 2024-02-19T22:08:08.612 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.610+0000 7f01437fe700 1 --2- 172.21.15.176:0/3604886513 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f013002fe10 0x7f01300322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:08.612 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.610+0000 7f014a410700 1 -- 172.21.15.176:0/3604886513 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0144059100 con 0x7f01440703f0 2024-02-19T22:08:08.612 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.611+0000 7f01437fe700 1 --2- 172.21.15.176:0/3604886513 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f013002fe10 0x7f01300322d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f01380096a0 tx=0x7f0138006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:08.614 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.614+0000 7f0140ff9700 1 -- 172.21.15.176:0/3604886513 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0134016070 con 0x7f01440703f0 2024-02-19T22:08:08.787 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:08.787+0000 7f014a410700 1 -- 172.21.15.176:0/3604886513 --> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi176", "addr": "172.21.15.176", "target": ["mon-mgr", ""]}) v1 -- 0x7f0144104b00 con 0x7f013002fe10 2024-02-19T22:08:09.382 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:09.382+0000 7f0140ff9700 1 -- 172.21.15.176:0/3604886513 <== mgr.14114 v2:172.21.15.176:6800/2184395941 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f0144104b00 con 0x7f013002fe10 2024-02-19T22:08:09.383 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: Added host 'smithi176' 2024-02-19T22:08:09.383 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:09.384+0000 7f014a410700 1 -- 172.21.15.176:0/3604886513 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f013002fe10 msgr2=0x7f01300322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:09.384 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:09.384+0000 7f014a410700 1 --2- 172.21.15.176:0/3604886513 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f013002fe10 0x7f01300322d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f01380096a0 tx=0x7f0138006b40).stop 2024-02-19T22:08:09.385 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:09.384+0000 7f014a410700 1 -- 172.21.15.176:0/3604886513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01440703f0 msgr2=0x7f01440fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:09.385 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:09.384+0000 7f014a410700 1 --2- 172.21.15.176:0/3604886513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01440703f0 0x7f01440fc510 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f0134011ed0 tx=0x7f0134006ee0).stop 2024-02-19T22:08:09.385 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:09.384+0000 7f014a410700 1 -- 172.21.15.176:0/3604886513 shutdown_connections 2024-02-19T22:08:09.385 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:09.384+0000 7f014a410700 1 --2- 172.21.15.176:0/3604886513 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f013002fe10 0x7f01300322d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:09.385 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:09.384+0000 7f014a410700 1 --2- 172.21.15.176:0/3604886513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01440703f0 0x7f01440fc510 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:09.385 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:09.384+0000 7f014a410700 1 -- 172.21.15.176:0/3604886513 >> 172.21.15.176:0/3604886513 conn(0x7f01440f6530 msgr2=0x7f01440f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:09.385 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:09.384+0000 7f014a410700 1 -- 172.21.15.176:0/3604886513 shutdown_connections 2024-02-19T22:08:09.385 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:09.384+0000 7f014a410700 1 -- 172.21.15.176:0/3604886513 wait complete. 2024-02-19T22:08:10.012 INFO:teuthology.orchestra.run.smithi176.stderr:Deploying mon service with default placement... 2024-02-19T22:08:10.012 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-02-19T22:08:10.527 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.527+0000 7f74540c7700 1 Processor -- start 2024-02-19T22:08:10.531 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.527+0000 7f74540c7700 1 -- start start 2024-02-19T22:08:10.531 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.527+0000 7f74540c7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f744c0fbea0 0x7f744c0fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:10.531 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.527+0000 7f74540c7700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c0fc7b0 con 0x7f744c0fbea0 2024-02-19T22:08:10.531 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.528+0000 7f7451e63700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f744c0fbea0 0x7f744c0fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:10.531 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.528+0000 7f7451e63700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f744c0fbea0 0x7f744c0fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:51362/0 (socket says 172.21.15.176:51362) 2024-02-19T22:08:10.531 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.528+0000 7f7451e63700 1 -- 172.21.15.176:0/3852574971 learned_addr learned my addr 172.21.15.176:0/3852574971 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:10.531 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.528+0000 7f7451e63700 1 -- 172.21.15.176:0/3852574971 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744c0fc8f0 con 0x7f744c0fbea0 2024-02-19T22:08:10.531 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.528+0000 7f7451e63700 1 --2- 172.21.15.176:0/3852574971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f744c0fbea0 0x7f744c0fc2c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f743c00b5f0 tx=0x7f743c006ee0).ready entity=mon.0 client_cookie=ff4c4974d5a7f2d4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:10.531 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.528+0000 7f7450e61700 1 -- 172.21.15.176:0/3852574971 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f743c00d600 con 0x7f744c0fbea0 2024-02-19T22:08:10.531 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.528+0000 7f7450e61700 1 -- 172.21.15.176:0/3852574971 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f743c00dbf0 con 0x7f744c0fbea0 2024-02-19T22:08:10.531 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.528+0000 7f7450e61700 1 -- 172.21.15.176:0/3852574971 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f743c011cb0 con 0x7f744c0fbea0 2024-02-19T22:08:10.531 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.529+0000 7f74540c7700 1 -- 172.21.15.176:0/3852574971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f744c0fbea0 msgr2=0x7f744c0fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:10.532 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.529+0000 7f74540c7700 1 --2- 172.21.15.176:0/3852574971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f744c0fbea0 0x7f744c0fc2c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f743c00b5f0 tx=0x7f743c006ee0).stop 2024-02-19T22:08:10.532 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.529+0000 7f74540c7700 1 -- 172.21.15.176:0/3852574971 shutdown_connections 2024-02-19T22:08:10.532 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.529+0000 7f74540c7700 1 --2- 172.21.15.176:0/3852574971 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f744c0fbea0 0x7f744c0fc2c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:10.532 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.529+0000 7f74540c7700 1 -- 172.21.15.176:0/3852574971 >> 172.21.15.176:0/3852574971 conn(0x7f744c0f76a0 msgr2=0x7f744c0f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:10.532 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.529+0000 7f74540c7700 1 -- 172.21.15.176:0/3852574971 shutdown_connections 2024-02-19T22:08:10.532 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.529+0000 7f74540c7700 1 -- 172.21.15.176:0/3852574971 wait complete. 2024-02-19T22:08:10.532 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.530+0000 7f74540c7700 1 Processor -- start 2024-02-19T22:08:10.532 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.530+0000 7f74540c7700 1 -- start start 2024-02-19T22:08:10.532 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.530+0000 7f74540c7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f744c0fbea0 0x7f744c10ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:10.532 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.530+0000 7f74540c7700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c10b080 con 0x7f744c0fbea0 2024-02-19T22:08:10.532 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.530+0000 7f7451e63700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f744c0fbea0 0x7f744c10ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:10.532 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.530+0000 7f7451e63700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f744c0fbea0 0x7f744c10ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:51366/0 (socket says 172.21.15.176:51366) 2024-02-19T22:08:10.532 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.530+0000 7f7451e63700 1 -- 172.21.15.176:0/1999988566 learned_addr learned my addr 172.21.15.176:0/1999988566 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:10.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.530+0000 7f7451e63700 1 -- 172.21.15.176:0/1999988566 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743c00b200 con 0x7f744c0fbea0 2024-02-19T22:08:10.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.530+0000 7f7451e63700 1 --2- 172.21.15.176:0/1999988566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f744c0fbea0 0x7f744c10ab90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f743c00bf40 tx=0x7f743c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:10.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.530+0000 7f743a7fc700 1 -- 172.21.15.176:0/1999988566 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f743c01b570 con 0x7f744c0fbea0 2024-02-19T22:08:10.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.530+0000 7f743a7fc700 1 -- 172.21.15.176:0/1999988566 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f743c01bb60 con 0x7f744c0fbea0 2024-02-19T22:08:10.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.530+0000 7f74540c7700 1 -- 172.21.15.176:0/1999988566 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744c10b280 con 0x7f744c0fbea0 2024-02-19T22:08:10.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.530+0000 7f743a7fc700 1 -- 172.21.15.176:0/1999988566 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f743c01aa20 con 0x7f744c0fbea0 2024-02-19T22:08:10.533 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.530+0000 7f74540c7700 1 -- 172.21.15.176:0/1999988566 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f744c10b710 con 0x7f744c0fbea0 2024-02-19T22:08:10.534 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.531+0000 7f743a7fc700 1 -- 172.21.15.176:0/1999988566 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f743c00f070 con 0x7f744c0fbea0 2024-02-19T22:08:10.534 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.531+0000 7f743a7fc700 1 --2- 172.21.15.176:0/1999988566 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f743402fe10 0x7f74340322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:10.534 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.531+0000 7f743a7fc700 1 -- 172.21.15.176:0/1999988566 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f743c0414d0 con 0x7f744c0fbea0 2024-02-19T22:08:10.534 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.531+0000 7f7451662700 1 --2- 172.21.15.176:0/1999988566 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f743402fe10 0x7f74340322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:10.534 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.532+0000 7f74540c7700 1 -- 172.21.15.176:0/1999988566 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f744c045ad0 con 0x7f744c0fbea0 2024-02-19T22:08:10.534 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.532+0000 7f7451662700 1 --2- 172.21.15.176:0/1999988566 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f743402fe10 0x7f74340322d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f74480096a0 tx=0x7f7448006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:10.536 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.536+0000 7f743a7fc700 1 -- 172.21.15.176:0/1999988566 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f743c016070 con 0x7f744c0fbea0 2024-02-19T22:08:10.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:10 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:08.788215+0000 mgr.smithi176.faoogw (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi176", "addr": "172.21.15.176", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:08:10.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:10 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:09.381980+0000 mon.smithi176 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:10.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:10 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:08:09.382265+0000 mgr.smithi176.faoogw (mgr.14114) 9 : cephadm [INF] Added host smithi176 2024-02-19T22:08:10.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:10 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:09.382493+0000 mon.smithi176 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:08:10.709 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.709+0000 7f74540c7700 1 -- 172.21.15.176:0/1999988566 --> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f744c104c80 con 0x7f743402fe10 2024-02-19T22:08:10.714 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.714+0000 7f743a7fc700 1 -- 172.21.15.176:0/1999988566 <== mgr.14114 v2:172.21.15.176:6800/2184395941 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f744c104c80 con 0x7f743402fe10 2024-02-19T22:08:10.714 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-19T22:08:10.716 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.716+0000 7f74540c7700 1 -- 172.21.15.176:0/1999988566 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f743402fe10 msgr2=0x7f74340322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:10.717 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.716+0000 7f74540c7700 1 --2- 172.21.15.176:0/1999988566 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f743402fe10 0x7f74340322d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f74480096a0 tx=0x7f7448006b40).stop 2024-02-19T22:08:10.717 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.716+0000 7f74540c7700 1 -- 172.21.15.176:0/1999988566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f744c0fbea0 msgr2=0x7f744c10ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:10.717 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.716+0000 7f74540c7700 1 --2- 172.21.15.176:0/1999988566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f744c0fbea0 0x7f744c10ab90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f743c00bf40 tx=0x7f743c006ee0).stop 2024-02-19T22:08:10.717 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.716+0000 7f74540c7700 1 -- 172.21.15.176:0/1999988566 shutdown_connections 2024-02-19T22:08:10.717 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.716+0000 7f74540c7700 1 --2- 172.21.15.176:0/1999988566 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f743402fe10 0x7f74340322d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:10.717 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.716+0000 7f74540c7700 1 --2- 172.21.15.176:0/1999988566 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f744c0fbea0 0x7f744c10ab90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:10.717 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.716+0000 7f74540c7700 1 -- 172.21.15.176:0/1999988566 >> 172.21.15.176:0/1999988566 conn(0x7f744c0f76a0 msgr2=0x7f744c0f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:10.717 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.716+0000 7f74540c7700 1 -- 172.21.15.176:0/1999988566 shutdown_connections 2024-02-19T22:08:10.717 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:10.716+0000 7f74540c7700 1 -- 172.21.15.176:0/1999988566 wait complete. 2024-02-19T22:08:11.277 INFO:teuthology.orchestra.run.smithi176.stderr:Deploying mgr service with default placement... 2024-02-19T22:08:11.277 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-02-19T22:08:11.839 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.788+0000 7f7e873ca700 1 Processor -- start 2024-02-19T22:08:11.839 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.788+0000 7f7e873ca700 1 -- start start 2024-02-19T22:08:11.839 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.789+0000 7f7e873ca700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e800703f0 0x7f7e80100e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:11.839 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.789+0000 7f7e873ca700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e800fc880 con 0x7f7e800703f0 2024-02-19T22:08:11.840 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.789+0000 7f7e85166700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e800703f0 0x7f7e80100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:11.840 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.789+0000 7f7e85166700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e800703f0 0x7f7e80100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:51380/0 (socket says 172.21.15.176:51380) 2024-02-19T22:08:11.840 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.789+0000 7f7e85166700 1 -- 172.21.15.176:0/3526117005 learned_addr learned my addr 172.21.15.176:0/3526117005 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:11.840 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.789+0000 7f7e85166700 1 -- 172.21.15.176:0/3526117005 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e80101380 con 0x7f7e800703f0 2024-02-19T22:08:11.840 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.789+0000 7f7e85166700 1 --2- 172.21.15.176:0/3526117005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e800703f0 0x7f7e80100e90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f7e7000b5f0 tx=0x7f7e70006ee0).ready entity=mon.0 client_cookie=7dc3460bc74c2cb7 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:11.840 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.789+0000 7f7e6ffff700 1 -- 172.21.15.176:0/3526117005 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e7000d600 con 0x7f7e800703f0 2024-02-19T22:08:11.840 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.789+0000 7f7e6ffff700 1 -- 172.21.15.176:0/3526117005 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7e7000dbf0 con 0x7f7e800703f0 2024-02-19T22:08:11.840 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.789+0000 7f7e6ffff700 1 -- 172.21.15.176:0/3526117005 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e70011ce0 con 0x7f7e800703f0 2024-02-19T22:08:11.840 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.790+0000 7f7e873ca700 1 -- 172.21.15.176:0/3526117005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e800703f0 msgr2=0x7f7e80100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:11.840 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.790+0000 7f7e873ca700 1 --2- 172.21.15.176:0/3526117005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e800703f0 0x7f7e80100e90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f7e7000b5f0 tx=0x7f7e70006ee0).stop 2024-02-19T22:08:11.840 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.790+0000 7f7e873ca700 1 -- 172.21.15.176:0/3526117005 shutdown_connections 2024-02-19T22:08:11.840 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.790+0000 7f7e873ca700 1 --2- 172.21.15.176:0/3526117005 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e800703f0 0x7f7e80100e90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:11.840 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.790+0000 7f7e873ca700 1 -- 172.21.15.176:0/3526117005 >> 172.21.15.176:0/3526117005 conn(0x7f7e800f6530 msgr2=0x7f7e800f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:11.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.790+0000 7f7e873ca700 1 -- 172.21.15.176:0/3526117005 shutdown_connections 2024-02-19T22:08:11.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.790+0000 7f7e873ca700 1 -- 172.21.15.176:0/3526117005 wait complete. 2024-02-19T22:08:11.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.791+0000 7f7e873ca700 1 Processor -- start 2024-02-19T22:08:11.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.791+0000 7f7e873ca700 1 -- start start 2024-02-19T22:08:11.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.791+0000 7f7e873ca700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e800703f0 0x7f7e8010cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:11.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.791+0000 7f7e873ca700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8010d290 con 0x7f7e800703f0 2024-02-19T22:08:11.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.791+0000 7f7e85166700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e800703f0 0x7f7e8010cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:11.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.791+0000 7f7e85166700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e800703f0 0x7f7e8010cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:51386/0 (socket says 172.21.15.176:51386) 2024-02-19T22:08:11.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.791+0000 7f7e85166700 1 -- 172.21.15.176:0/1753530825 learned_addr learned my addr 172.21.15.176:0/1753530825 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:11.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.791+0000 7f7e85166700 1 -- 172.21.15.176:0/1753530825 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7000b200 con 0x7f7e800703f0 2024-02-19T22:08:11.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.791+0000 7f7e85166700 1 --2- 172.21.15.176:0/1753530825 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e800703f0 0x7f7e8010cda0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f7e70011ed0 tx=0x7f7e70006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:11.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.791+0000 7f7e6dffb700 1 -- 172.21.15.176:0/1753530825 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e7001b570 con 0x7f7e800703f0 2024-02-19T22:08:11.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.791+0000 7f7e6dffb700 1 -- 172.21.15.176:0/1753530825 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7e7001bb60 con 0x7f7e800703f0 2024-02-19T22:08:11.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.791+0000 7f7e873ca700 1 -- 172.21.15.176:0/1753530825 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8010d490 con 0x7f7e800703f0 2024-02-19T22:08:11.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.791+0000 7f7e6dffb700 1 -- 172.21.15.176:0/1753530825 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e7001aad0 con 0x7f7e800703f0 2024-02-19T22:08:11.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.792+0000 7f7e873ca700 1 -- 172.21.15.176:0/1753530825 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8010e0d0 con 0x7f7e800703f0 2024-02-19T22:08:11.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.792+0000 7f7e6dffb700 1 -- 172.21.15.176:0/1753530825 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f7e7000f070 con 0x7f7e800703f0 2024-02-19T22:08:11.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.793+0000 7f7e6dffb700 1 --2- 172.21.15.176:0/1753530825 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f7e6802fe70 0x7f7e68032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:11.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.793+0000 7f7e6dffb700 1 -- 172.21.15.176:0/1753530825 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7e70041340 con 0x7f7e800703f0 2024-02-19T22:08:11.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.793+0000 7f7e873ca700 1 -- 172.21.15.176:0/1753530825 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8010d920 con 0x7f7e800703f0 2024-02-19T22:08:11.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.793+0000 7f7e84965700 1 --2- 172.21.15.176:0/1753530825 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f7e6802fe70 0x7f7e68032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:11.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.793+0000 7f7e84965700 1 --2- 172.21.15.176:0/1753530825 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f7e6802fe70 0x7f7e68032330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f7e7c0096a0 tx=0x7f7e7c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:11.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.797+0000 7f7e6dffb700 1 -- 172.21.15.176:0/1753530825 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e70016070 con 0x7f7e800703f0 2024-02-19T22:08:11.972 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.972+0000 7f7e873ca700 1 -- 172.21.15.176:0/1753530825 --> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e80102180 con 0x7f7e6802fe70 2024-02-19T22:08:11.976 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.976+0000 7f7e6dffb700 1 -- 172.21.15.176:0/1753530825 <== mgr.14114 v2:172.21.15.176:6800/2184395941 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7e80102180 con 0x7f7e6802fe70 2024-02-19T22:08:11.976 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-19T22:08:11.979 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.978+0000 7f7e873ca700 1 -- 172.21.15.176:0/1753530825 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f7e6802fe70 msgr2=0x7f7e68032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:11.979 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.978+0000 7f7e873ca700 1 --2- 172.21.15.176:0/1753530825 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f7e6802fe70 0x7f7e68032330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f7e7c0096a0 tx=0x7f7e7c006b40).stop 2024-02-19T22:08:11.979 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.978+0000 7f7e873ca700 1 -- 172.21.15.176:0/1753530825 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e800703f0 msgr2=0x7f7e8010cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:11.979 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.978+0000 7f7e873ca700 1 --2- 172.21.15.176:0/1753530825 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e800703f0 0x7f7e8010cda0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f7e70011ed0 tx=0x7f7e70006ee0).stop 2024-02-19T22:08:11.979 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.978+0000 7f7e873ca700 1 -- 172.21.15.176:0/1753530825 shutdown_connections 2024-02-19T22:08:11.979 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.978+0000 7f7e873ca700 1 --2- 172.21.15.176:0/1753530825 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f7e6802fe70 0x7f7e68032330 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:11.979 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.978+0000 7f7e873ca700 1 --2- 172.21.15.176:0/1753530825 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e800703f0 0x7f7e8010cda0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:11.979 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.978+0000 7f7e873ca700 1 -- 172.21.15.176:0/1753530825 >> 172.21.15.176:0/1753530825 conn(0x7f7e800f6530 msgr2=0x7f7e800f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:11.979 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.978+0000 7f7e873ca700 1 -- 172.21.15.176:0/1753530825 shutdown_connections 2024-02-19T22:08:11.979 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:11.978+0000 7f7e873ca700 1 -- 172.21.15.176:0/1753530825 wait complete. 2024-02-19T22:08:12.005 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:10.710859+0000 mgr.smithi176.faoogw (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:08:12.005 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:08:10.711568+0000 mgr.smithi176.faoogw (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-19T22:08:12.005 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:10.714612+0000 mon.smithi176 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:12.641 INFO:teuthology.orchestra.run.smithi176.stderr:Deploying crash service with default placement... 2024-02-19T22:08:12.641 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-02-19T22:08:13.385 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:12 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:11.973497+0000 mgr.smithi176.faoogw (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:08:13.385 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:12 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:08:11.973993+0000 mgr.smithi176.faoogw (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-19T22:08:13.385 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:12 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:11.977022+0000 mon.smithi176 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:13.492 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.419+0000 7eff9c469700 1 Processor -- start 2024-02-19T22:08:13.492 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.419+0000 7eff9c469700 1 -- start start 2024-02-19T22:08:13.493 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.419+0000 7eff9c469700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff94070370 0x7eff940feb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:13.493 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.419+0000 7eff9c469700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff94070790 con 0x7eff94070370 2024-02-19T22:08:13.493 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.419+0000 7eff9a205700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff94070370 0x7eff940feb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:13.493 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.420+0000 7eff9a205700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff94070370 0x7eff940feb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:51396/0 (socket says 172.21.15.176:51396) 2024-02-19T22:08:13.493 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.420+0000 7eff9a205700 1 -- 172.21.15.176:0/1315565501 learned_addr learned my addr 172.21.15.176:0/1315565501 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:13.493 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.420+0000 7eff9a205700 1 -- 172.21.15.176:0/1315565501 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff940ff000 con 0x7eff94070370 2024-02-19T22:08:13.493 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.420+0000 7eff9a205700 1 --2- 172.21.15.176:0/1315565501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff94070370 0x7eff940feb10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7eff8c00b5f0 tx=0x7eff8c006ee0).ready entity=mon.0 client_cookie=af15dc334c02c22c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:13.493 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.420+0000 7eff99203700 1 -- 172.21.15.176:0/1315565501 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff8c00d600 con 0x7eff94070370 2024-02-19T22:08:13.493 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.420+0000 7eff99203700 1 -- 172.21.15.176:0/1315565501 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7eff8c00dbf0 con 0x7eff94070370 2024-02-19T22:08:13.494 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.420+0000 7eff99203700 1 -- 172.21.15.176:0/1315565501 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff8c011cb0 con 0x7eff94070370 2024-02-19T22:08:13.494 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.421+0000 7eff9c469700 1 -- 172.21.15.176:0/1315565501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff94070370 msgr2=0x7eff940feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:13.494 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.421+0000 7eff9c469700 1 --2- 172.21.15.176:0/1315565501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff94070370 0x7eff940feb10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7eff8c00b5f0 tx=0x7eff8c006ee0).stop 2024-02-19T22:08:13.494 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.421+0000 7eff9c469700 1 -- 172.21.15.176:0/1315565501 shutdown_connections 2024-02-19T22:08:13.494 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.421+0000 7eff9c469700 1 --2- 172.21.15.176:0/1315565501 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff94070370 0x7eff940feb10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:13.494 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.421+0000 7eff9c469700 1 -- 172.21.15.176:0/1315565501 >> 172.21.15.176:0/1315565501 conn(0x7eff940f6530 msgr2=0x7eff940f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:13.495 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.421+0000 7eff9c469700 1 -- 172.21.15.176:0/1315565501 shutdown_connections 2024-02-19T22:08:13.495 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.421+0000 7eff9c469700 1 -- 172.21.15.176:0/1315565501 wait complete. 2024-02-19T22:08:13.495 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.421+0000 7eff9c469700 1 Processor -- start 2024-02-19T22:08:13.495 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.421+0000 7eff9c469700 1 -- start start 2024-02-19T22:08:13.495 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.422+0000 7eff9c469700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff94070370 0x7eff940fc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:13.496 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.422+0000 7eff9c469700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff940faa10 con 0x7eff94070370 2024-02-19T22:08:13.496 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.422+0000 7eff9a205700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff94070370 0x7eff940fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:13.496 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.422+0000 7eff9a205700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff94070370 0x7eff940fc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:51404/0 (socket says 172.21.15.176:51404) 2024-02-19T22:08:13.496 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.422+0000 7eff9a205700 1 -- 172.21.15.176:0/3641535460 learned_addr learned my addr 172.21.15.176:0/3641535460 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:13.496 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.422+0000 7eff9a205700 1 -- 172.21.15.176:0/3641535460 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff8c00b200 con 0x7eff94070370 2024-02-19T22:08:13.496 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.422+0000 7eff9a205700 1 --2- 172.21.15.176:0/3641535460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff94070370 0x7eff940fc590 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7eff8c006800 tx=0x7eff8c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:13.497 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.422+0000 7eff92ffd700 1 -- 172.21.15.176:0/3641535460 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff8c01b570 con 0x7eff94070370 2024-02-19T22:08:13.497 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.422+0000 7eff92ffd700 1 -- 172.21.15.176:0/3641535460 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7eff8c01bb60 con 0x7eff94070370 2024-02-19T22:08:13.497 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.422+0000 7eff9c469700 1 -- 172.21.15.176:0/3641535460 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff940fabb0 con 0x7eff94070370 2024-02-19T22:08:13.497 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.422+0000 7eff92ffd700 1 -- 172.21.15.176:0/3641535460 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff8c01a9f0 con 0x7eff94070370 2024-02-19T22:08:13.497 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.422+0000 7eff9c469700 1 -- 172.21.15.176:0/3641535460 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff940fb040 con 0x7eff94070370 2024-02-19T22:08:13.497 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.423+0000 7eff92ffd700 1 -- 172.21.15.176:0/3641535460 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7eff8c00f070 con 0x7eff94070370 2024-02-19T22:08:13.497 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.423+0000 7eff92ffd700 1 --2- 172.21.15.176:0/3641535460 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7eff8002fe10 0x7eff800322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:13.498 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.423+0000 7eff92ffd700 1 -- 172.21.15.176:0/3641535460 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7eff8c0414d0 con 0x7eff94070370 2024-02-19T22:08:13.498 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.423+0000 7eff9c469700 1 -- 172.21.15.176:0/3641535460 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff94059100 con 0x7eff94070370 2024-02-19T22:08:13.498 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.423+0000 7eff99a04700 1 --2- 172.21.15.176:0/3641535460 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7eff8002fe10 0x7eff800322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:13.498 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.424+0000 7eff99a04700 1 --2- 172.21.15.176:0/3641535460 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7eff8002fe10 0x7eff800322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7eff880096a0 tx=0x7eff88006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:13.498 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.427+0000 7eff92ffd700 1 -- 172.21.15.176:0/3641535460 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff8c016070 con 0x7eff94070370 2024-02-19T22:08:13.606 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.606+0000 7eff9c469700 1 -- 172.21.15.176:0/3641535460 --> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7eff94104af0 con 0x7eff8002fe10 2024-02-19T22:08:13.610 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.610+0000 7eff92ffd700 1 -- 172.21.15.176:0/3641535460 <== mgr.14114 v2:172.21.15.176:6800/2184395941 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7eff94104af0 con 0x7eff8002fe10 2024-02-19T22:08:13.610 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-19T22:08:13.613 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.611+0000 7eff9c469700 1 -- 172.21.15.176:0/3641535460 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7eff8002fe10 msgr2=0x7eff800322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:13.614 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.611+0000 7eff9c469700 1 --2- 172.21.15.176:0/3641535460 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7eff8002fe10 0x7eff800322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7eff880096a0 tx=0x7eff88006b40).stop 2024-02-19T22:08:13.614 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.611+0000 7eff9c469700 1 -- 172.21.15.176:0/3641535460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff94070370 msgr2=0x7eff940fc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:13.614 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.611+0000 7eff9c469700 1 --2- 172.21.15.176:0/3641535460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff94070370 0x7eff940fc590 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7eff8c006800 tx=0x7eff8c006ee0).stop 2024-02-19T22:08:13.614 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.612+0000 7eff9c469700 1 -- 172.21.15.176:0/3641535460 shutdown_connections 2024-02-19T22:08:13.614 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.612+0000 7eff9c469700 1 --2- 172.21.15.176:0/3641535460 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7eff8002fe10 0x7eff800322d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:13.615 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.612+0000 7eff9c469700 1 --2- 172.21.15.176:0/3641535460 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7eff94070370 0x7eff940fc590 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:13.616 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.612+0000 7eff9c469700 1 -- 172.21.15.176:0/3641535460 >> 172.21.15.176:0/3641535460 conn(0x7eff940f6530 msgr2=0x7eff940f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:13.616 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.612+0000 7eff9c469700 1 -- 172.21.15.176:0/3641535460 shutdown_connections 2024-02-19T22:08:13.616 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:13.612+0000 7eff9c469700 1 -- 172.21.15.176:0/3641535460 wait complete. 2024-02-19T22:08:14.199 INFO:teuthology.orchestra.run.smithi176.stderr:Enabling mgr prometheus module... 2024-02-19T22:08:14.199 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-02-19T22:08:14.711 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.695+0000 7f242a082700 1 Processor -- start 2024-02-19T22:08:14.711 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.695+0000 7f242a082700 1 -- start start 2024-02-19T22:08:14.711 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.696+0000 7f242a082700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24240fad90 0x7f24240fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:14.711 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.696+0000 7f242a082700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24240fb6a0 con 0x7f24240fad90 2024-02-19T22:08:14.711 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.696+0000 7f24237fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24240fad90 0x7f24240fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:14.711 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.696+0000 7f24237fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24240fad90 0x7f24240fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38148/0 (socket says 172.21.15.176:38148) 2024-02-19T22:08:14.712 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.696+0000 7f24237fe700 1 -- 172.21.15.176:0/2138810255 learned_addr learned my addr 172.21.15.176:0/2138810255 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:14.712 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.696+0000 7f24237fe700 1 -- 172.21.15.176:0/2138810255 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24240fb7e0 con 0x7f24240fad90 2024-02-19T22:08:14.712 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.696+0000 7f24237fe700 1 --2- 172.21.15.176:0/2138810255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24240fad90 0x7f24240fb1b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f240c00b5f0 tx=0x7f240c006ee0).ready entity=mon.0 client_cookie=7dfaa0bdaab57a64 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:14.712 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.696+0000 7f24227fc700 1 -- 172.21.15.176:0/2138810255 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f240c00d600 con 0x7f24240fad90 2024-02-19T22:08:14.712 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.696+0000 7f24227fc700 1 -- 172.21.15.176:0/2138810255 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f240c00dbf0 con 0x7f24240fad90 2024-02-19T22:08:14.712 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.696+0000 7f24227fc700 1 -- 172.21.15.176:0/2138810255 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f240c011cb0 con 0x7f24240fad90 2024-02-19T22:08:14.712 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.697+0000 7f242a082700 1 -- 172.21.15.176:0/2138810255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24240fad90 msgr2=0x7f24240fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:14.712 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.697+0000 7f242a082700 1 --2- 172.21.15.176:0/2138810255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24240fad90 0x7f24240fb1b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f240c00b5f0 tx=0x7f240c006ee0).stop 2024-02-19T22:08:14.712 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.697+0000 7f242a082700 1 -- 172.21.15.176:0/2138810255 shutdown_connections 2024-02-19T22:08:14.712 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.697+0000 7f242a082700 1 --2- 172.21.15.176:0/2138810255 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24240fad90 0x7f24240fb1b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:14.712 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.697+0000 7f242a082700 1 -- 172.21.15.176:0/2138810255 >> 172.21.15.176:0/2138810255 conn(0x7f24240f6530 msgr2=0x7f24240f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:14.712 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.697+0000 7f242a082700 1 -- 172.21.15.176:0/2138810255 shutdown_connections 2024-02-19T22:08:14.713 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.697+0000 7f242a082700 1 -- 172.21.15.176:0/2138810255 wait complete. 2024-02-19T22:08:14.713 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.697+0000 7f242a082700 1 Processor -- start 2024-02-19T22:08:14.713 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.698+0000 7f242a082700 1 -- start start 2024-02-19T22:08:14.713 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.698+0000 7f242a082700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24240fad90 0x7f2424100fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:14.713 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.698+0000 7f242a082700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24240ff4c0 con 0x7f24240fad90 2024-02-19T22:08:14.713 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.698+0000 7f24237fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24240fad90 0x7f2424100fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:14.713 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.698+0000 7f24237fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24240fad90 0x7f2424100fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38162/0 (socket says 172.21.15.176:38162) 2024-02-19T22:08:14.713 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.698+0000 7f24237fe700 1 -- 172.21.15.176:0/3385508435 learned_addr learned my addr 172.21.15.176:0/3385508435 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:14.713 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.698+0000 7f24237fe700 1 -- 172.21.15.176:0/3385508435 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c00b200 con 0x7f24240fad90 2024-02-19T22:08:14.713 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.698+0000 7f24237fe700 1 --2- 172.21.15.176:0/3385508435 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24240fad90 0x7f2424100fd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f240c011ea0 tx=0x7f240c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:14.714 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.698+0000 7f241bfff700 1 -- 172.21.15.176:0/3385508435 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f240c01b570 con 0x7f24240fad90 2024-02-19T22:08:14.714 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.698+0000 7f241bfff700 1 -- 172.21.15.176:0/3385508435 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f240c01bb60 con 0x7f24240fad90 2024-02-19T22:08:14.714 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.698+0000 7f242a082700 1 -- 172.21.15.176:0/3385508435 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24240ff6c0 con 0x7f24240fad90 2024-02-19T22:08:14.714 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.698+0000 7f241bfff700 1 -- 172.21.15.176:0/3385508435 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f240c01a8d0 con 0x7f24240fad90 2024-02-19T22:08:14.714 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.698+0000 7f242a082700 1 -- 172.21.15.176:0/3385508435 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24240ffb50 con 0x7f24240fad90 2024-02-19T22:08:14.714 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.699+0000 7f241bfff700 1 -- 172.21.15.176:0/3385508435 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f240c00f070 con 0x7f24240fad90 2024-02-19T22:08:14.714 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.699+0000 7f241bfff700 1 --2- 172.21.15.176:0/3385508435 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f240402fe70 0x7f2404032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:14.714 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.699+0000 7f242a082700 1 -- 172.21.15.176:0/3385508435 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2424045ad0 con 0x7f24240fad90 2024-02-19T22:08:14.714 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.699+0000 7f241bfff700 1 -- 172.21.15.176:0/3385508435 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f240c0413e0 con 0x7f24240fad90 2024-02-19T22:08:14.715 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.701+0000 7f2422ffd700 1 --2- 172.21.15.176:0/3385508435 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f240402fe70 0x7f2404032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:14.715 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.701+0000 7f2422ffd700 1 --2- 172.21.15.176:0/3385508435 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f240402fe70 0x7f2404032330 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f24140096a0 tx=0x7f2414006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:14.715 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.703+0000 7f241bfff700 1 -- 172.21.15.176:0/3385508435 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f240c016020 con 0x7f24240fad90 2024-02-19T22:08:14.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:14 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:13.607437+0000 mgr.smithi176.faoogw (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:08:14.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:14 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:08:13.608099+0000 mgr.smithi176.faoogw (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-19T22:08:14.887 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:14 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:13.610435+0000 mon.smithi176 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.176:0/292190697' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:14.889 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:14.889+0000 7f242a082700 1 -- 172.21.15.176:0/3385508435 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f2424027a00 con 0x7f24240fad90 2024-02-19T22:08:15.720 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:15.719+0000 7f241bfff700 1 -- 172.21.15.176:0/3385508435 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f240c022d80 con 0x7f24240fad90 2024-02-19T22:08:15.721 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:15.721+0000 7f241bfff700 1 -- 172.21.15.176:0/3385508435 <== mon.0 v2:172.21.15.176:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f240c0224f0 con 0x7f24240fad90 2024-02-19T22:08:15.723 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:15.723+0000 7f242a082700 1 -- 172.21.15.176:0/3385508435 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f240402fe70 msgr2=0x7f2404032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:15.723 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:15.723+0000 7f242a082700 1 --2- 172.21.15.176:0/3385508435 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f240402fe70 0x7f2404032330 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f24140096a0 tx=0x7f2414006b40).stop 2024-02-19T22:08:15.724 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:15.723+0000 7f242a082700 1 -- 172.21.15.176:0/3385508435 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24240fad90 msgr2=0x7f2424100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:15.724 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:15.723+0000 7f242a082700 1 --2- 172.21.15.176:0/3385508435 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24240fad90 0x7f2424100fd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f240c011ea0 tx=0x7f240c006ee0).stop 2024-02-19T22:08:15.724 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:15.723+0000 7f242a082700 1 -- 172.21.15.176:0/3385508435 shutdown_connections 2024-02-19T22:08:15.724 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:15.723+0000 7f242a082700 1 --2- 172.21.15.176:0/3385508435 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f240402fe70 0x7f2404032330 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:15.724 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:15.723+0000 7f242a082700 1 --2- 172.21.15.176:0/3385508435 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f24240fad90 0x7f2424100fd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:15.724 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:15.723+0000 7f242a082700 1 -- 172.21.15.176:0/3385508435 >> 172.21.15.176:0/3385508435 conn(0x7f24240f6530 msgr2=0x7f2424070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:15.724 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:15.724+0000 7f242a082700 1 -- 172.21.15.176:0/3385508435 shutdown_connections 2024-02-19T22:08:15.724 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:15.724+0000 7f242a082700 1 -- 172.21.15.176:0/3385508435 wait complete. 2024-02-19T22:08:16.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:14.889979+0000 mon.smithi176 (mon.0) 59 : audit [INF] from='client.? 172.21.15.176:0/3385508435' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-19T22:08:16.437 INFO:teuthology.orchestra.run.smithi176.stderr:Deploying prometheus service with default placement... 2024-02-19T22:08:16.438 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-02-19T22:08:16.958 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.941+0000 7f701ed8e700 1 Processor -- start 2024-02-19T22:08:16.958 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.941+0000 7f701ed8e700 1 -- start start 2024-02-19T22:08:16.958 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.941+0000 7f701ed8e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70180fae00 0x7f70180fb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:16.958 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.941+0000 7f701ed8e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70180fb710 con 0x7f70180fae00 2024-02-19T22:08:16.958 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.942+0000 7f701cb2a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70180fae00 0x7f70180fb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:16.958 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.943+0000 7f701cb2a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70180fae00 0x7f70180fb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38190/0 (socket says 172.21.15.176:38190) 2024-02-19T22:08:16.958 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.943+0000 7f701cb2a700 1 -- 172.21.15.176:0/4236944000 learned_addr learned my addr 172.21.15.176:0/4236944000 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:16.958 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.943+0000 7f701cb2a700 1 -- 172.21.15.176:0/4236944000 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70180fb850 con 0x7f70180fae00 2024-02-19T22:08:16.958 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.943+0000 7f701cb2a700 1 --2- 172.21.15.176:0/4236944000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70180fae00 0x7f70180fb220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f700c00b5f0 tx=0x7f700c006ee0).ready entity=mon.0 client_cookie=2d50739a4469ea4b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:16.959 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.943+0000 7f70177fe700 1 -- 172.21.15.176:0/4236944000 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f700c00d600 con 0x7f70180fae00 2024-02-19T22:08:16.959 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.943+0000 7f70177fe700 1 -- 172.21.15.176:0/4236944000 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f700c00dbf0 con 0x7f70180fae00 2024-02-19T22:08:16.959 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.943+0000 7f70177fe700 1 -- 172.21.15.176:0/4236944000 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f700c011cb0 con 0x7f70180fae00 2024-02-19T22:08:16.959 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.944+0000 7f701ed8e700 1 -- 172.21.15.176:0/4236944000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70180fae00 msgr2=0x7f70180fb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:16.959 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.944+0000 7f701ed8e700 1 --2- 172.21.15.176:0/4236944000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70180fae00 0x7f70180fb220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f700c00b5f0 tx=0x7f700c006ee0).stop 2024-02-19T22:08:16.959 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.944+0000 7f701ed8e700 1 -- 172.21.15.176:0/4236944000 shutdown_connections 2024-02-19T22:08:16.959 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.944+0000 7f701ed8e700 1 --2- 172.21.15.176:0/4236944000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70180fae00 0x7f70180fb220 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:16.959 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.944+0000 7f701ed8e700 1 -- 172.21.15.176:0/4236944000 >> 172.21.15.176:0/4236944000 conn(0x7f70180f65b0 msgr2=0x7f70180f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:16.959 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.944+0000 7f701ed8e700 1 -- 172.21.15.176:0/4236944000 shutdown_connections 2024-02-19T22:08:16.959 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.944+0000 7f701ed8e700 1 -- 172.21.15.176:0/4236944000 wait complete. 2024-02-19T22:08:16.959 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.945+0000 7f701ed8e700 1 Processor -- start 2024-02-19T22:08:16.959 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.945+0000 7f701ed8e700 1 -- start start 2024-02-19T22:08:16.959 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.945+0000 7f701ed8e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70180fae00 0x7f7018109ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:16.959 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.945+0000 7f701ed8e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7018109fb0 con 0x7f70180fae00 2024-02-19T22:08:16.960 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.946+0000 7f701cb2a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70180fae00 0x7f7018109ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:16.960 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.946+0000 7f701cb2a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70180fae00 0x7f7018109ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38194/0 (socket says 172.21.15.176:38194) 2024-02-19T22:08:16.960 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.946+0000 7f701cb2a700 1 -- 172.21.15.176:0/2046714281 learned_addr learned my addr 172.21.15.176:0/2046714281 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:16.965 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.946+0000 7f701cb2a700 1 -- 172.21.15.176:0/2046714281 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700c00b200 con 0x7f70180fae00 2024-02-19T22:08:16.965 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.946+0000 7f701cb2a700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70180fae00 0x7f7018109ac0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f700c00dfc0 tx=0x7f700c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:16.965 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.946+0000 7f70157fa700 1 -- 172.21.15.176:0/2046714281 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f700c01b410 con 0x7f70180fae00 2024-02-19T22:08:16.965 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.947+0000 7f70157fa700 1 -- 172.21.15.176:0/2046714281 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f700c01ba00 con 0x7f70180fae00 2024-02-19T22:08:16.965 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.947+0000 7f70157fa700 1 -- 172.21.15.176:0/2046714281 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f700c01a990 con 0x7f70180fae00 2024-02-19T22:08:16.966 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.947+0000 7f701ed8e700 1 -- 172.21.15.176:0/2046714281 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701810a1b0 con 0x7f70180fae00 2024-02-19T22:08:16.966 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.947+0000 7f701ed8e700 1 -- 172.21.15.176:0/2046714281 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701810a640 con 0x7f70180fae00 2024-02-19T22:08:16.966 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.949+0000 7f70157fa700 1 -- 172.21.15.176:0/2046714281 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f700c00f070 con 0x7f70180fae00 2024-02-19T22:08:16.966 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.950+0000 7f70157fa700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f6ffc02fec0 0x7f6ffc032380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:16.966 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.950+0000 7f7017fff700 1 -- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f6ffc02fec0 msgr2=0x7f6ffc032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/2184395941 2024-02-19T22:08:16.966 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.950+0000 7f7017fff700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f6ffc02fec0 0x7f6ffc032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:08:16.966 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.950+0000 7f70157fa700 1 -- 172.21.15.176:0/2046714281 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f700c0182e0 con 0x7f70180fae00 2024-02-19T22:08:16.966 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.950+0000 7f701ed8e700 1 -- 172.21.15.176:0/2046714281 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7018045ad0 con 0x7f70180fae00 2024-02-19T22:08:16.966 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:16.958+0000 7f70157fa700 1 -- 172.21.15.176:0/2046714281 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f700c016020 con 0x7f70180fae00 2024-02-19T22:08:17.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:16 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:15.721761+0000 mon.smithi176 (mon.0) 60 : audit [INF] from='client.? 172.21.15.176:0/3385508435' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-19T22:08:17.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:16 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:15.721858+0000 mon.smithi176 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi176.faoogw(active, since 19s) 2024-02-19T22:08:17.151 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:17.151+0000 7f7017fff700 1 -- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f6ffc02fec0 msgr2=0x7f6ffc032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/2184395941 2024-02-19T22:08:17.151 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:17.151+0000 7f7017fff700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f6ffc02fec0 0x7f6ffc032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:08:17.164 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:17.164+0000 7f701ed8e700 1 -- 172.21.15.176:0/2046714281 --> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f7018103b40 con 0x7f6ffc02fec0 2024-02-19T22:08:17.552 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:17.552+0000 7f7017fff700 1 -- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f6ffc02fec0 msgr2=0x7f6ffc032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/2184395941 2024-02-19T22:08:17.552 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:17.552+0000 7f7017fff700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f6ffc02fec0 0x7f6ffc032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T22:08:18.353 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:18.353+0000 7f7017fff700 1 -- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f6ffc02fec0 msgr2=0x7f6ffc032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/2184395941 2024-02-19T22:08:18.353 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:18.353+0000 7f7017fff700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f6ffc02fec0 0x7f6ffc032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T22:08:19.955 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:19.954+0000 7f7017fff700 1 -- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f6ffc02fec0 msgr2=0x7f6ffc032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/2184395941 2024-02-19T22:08:19.955 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:19.954+0000 7f7017fff700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f6ffc02fec0 0x7f6ffc032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-19T22:08:20.718 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:20.717+0000 7f70157fa700 1 -- 172.21.15.176:0/2046714281 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f700c01bb70 con 0x7f70180fae00 2024-02-19T22:08:20.718 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:20.718+0000 7f70157fa700 1 -- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f6ffc02fec0 msgr2=0x7f6ffc032380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:08:20.718 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:20.718+0000 7f70157fa700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f6ffc02fec0 0x7f6ffc032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:08:21.719 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.718+0000 7f70157fa700 1 -- 172.21.15.176:0/2046714281 <== mon.0 v2:172.21.15.176:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f700c01b570 con 0x7f70180fae00 2024-02-19T22:08:21.720 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.719+0000 7f70157fa700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f6ffc035ef0 0x7f6ffc0382e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:21.720 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.719+0000 7f70157fa700 1 -- 172.21.15.176:0/2046714281 --> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f7018103b40 con 0x7f6ffc035ef0 2024-02-19T22:08:21.720 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.719+0000 7f7017fff700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f6ffc035ef0 0x7f6ffc0382e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:21.720 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.719+0000 7f7017fff700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f6ffc035ef0 0x7f6ffc0382e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f70040036d0 tx=0x7f7004006ca0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:21.723 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.723+0000 7f70157fa700 1 -- 172.21.15.176:0/2046714281 <== mgr.14142 v2:172.21.15.176:6800/918424985 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f7018103b40 con 0x7f6ffc035ef0 2024-02-19T22:08:21.723 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-19T22:08:21.726 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.726+0000 7f701ed8e700 1 -- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f6ffc035ef0 msgr2=0x7f6ffc0382e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:21.726 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.726+0000 7f701ed8e700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f6ffc035ef0 0x7f6ffc0382e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f70040036d0 tx=0x7f7004006ca0).stop 2024-02-19T22:08:21.727 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.726+0000 7f701ed8e700 1 -- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70180fae00 msgr2=0x7f7018109ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:21.727 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.726+0000 7f701ed8e700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70180fae00 0x7f7018109ac0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f700c00dfc0 tx=0x7f700c006ee0).stop 2024-02-19T22:08:21.727 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.726+0000 7f701ed8e700 1 -- 172.21.15.176:0/2046714281 shutdown_connections 2024-02-19T22:08:21.727 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.726+0000 7f701ed8e700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f6ffc035ef0 0x7f6ffc0382e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:21.727 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.726+0000 7f701ed8e700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f70180fae00 0x7f7018109ac0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:21.727 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.726+0000 7f701ed8e700 1 --2- 172.21.15.176:0/2046714281 >> [v2:172.21.15.176:6800/2184395941,v1:172.21.15.176:6801/2184395941] conn(0x7f6ffc02fec0 0x7f6ffc032380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:08:21.727 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.726+0000 7f701ed8e700 1 -- 172.21.15.176:0/2046714281 >> 172.21.15.176:0/2046714281 conn(0x7f70180f65b0 msgr2=0x7f70180f7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:21.727 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.726+0000 7f701ed8e700 1 -- 172.21.15.176:0/2046714281 shutdown_connections 2024-02-19T22:08:21.727 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:21.726+0000 7f701ed8e700 1 -- 172.21.15.176:0/2046714281 wait complete. 2024-02-19T22:08:22.085 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:20.667629+0000 mon.smithi176 (mon.0) 62 : cluster [INF] Active manager daemon smithi176.faoogw restarted 2024-02-19T22:08:22.085 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:20.668046+0000 mon.smithi176 (mon.0) 63 : cluster [INF] Activating manager daemon smithi176.faoogw 2024-02-19T22:08:22.085 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:20.672467+0000 mon.smithi176 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-19T22:08:22.085 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:20.719388+0000 mon.smithi176 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi176.faoogw(active, starting, since 0.051432s) 2024-02-19T22:08:22.085 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:20.720399+0000 mon.smithi176 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-02-19T22:08:22.085 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:20.720988+0000 mon.smithi176 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mgr metadata", "who": "smithi176.faoogw", "id": "smithi176.faoogw"}]: dispatch 2024-02-19T22:08:22.085 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:20.721637+0000 mon.smithi176 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:08:22.085 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:20.721802+0000 mon.smithi176 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:08:22.085 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:20.721923+0000 mon.smithi176 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:08:22.085 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:20.735837+0000 mon.smithi176 (mon.0) 71 : cluster [INF] Manager daemon smithi176.faoogw is now available 2024-02-19T22:08:22.086 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:20.833054+0000 mon.smithi176 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:08:22.086 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:20.834410+0000 mon.smithi176 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:08:22.086 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:20.850018+0000 mon.smithi176 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:08:22.086 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:20.858369+0000 mon.smithi176 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:08:22.353 INFO:teuthology.orchestra.run.smithi176.stderr:Deploying grafana service with default placement... 2024-02-19T22:08:22.354 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-02-19T22:08:22.909 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.905+0000 7f474f786700 1 Processor -- start 2024-02-19T22:08:22.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.905+0000 7f474f786700 1 -- start start 2024-02-19T22:08:22.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.905+0000 7f474f786700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4748072520 0x7f4748070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:22.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.905+0000 7f474f786700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4748072940 con 0x7f4748072520 2024-02-19T22:08:22.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.906+0000 7f474d522700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4748072520 0x7f4748070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:22.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.906+0000 7f474d522700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4748072520 0x7f4748070ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38260/0 (socket says 172.21.15.176:38260) 2024-02-19T22:08:22.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.906+0000 7f474d522700 1 -- 172.21.15.176:0/3847628679 learned_addr learned my addr 172.21.15.176:0/3847628679 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:22.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.906+0000 7f474d522700 1 -- 172.21.15.176:0/3847628679 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4748070fb0 con 0x7f4748072520 2024-02-19T22:08:22.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.906+0000 7f474d522700 1 --2- 172.21.15.176:0/3847628679 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4748072520 0x7f4748070ac0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f473800b5f0 tx=0x7f4738006ee0).ready entity=mon.0 client_cookie=458800bc31b1b843 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:22.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.906+0000 7f4737fff700 1 -- 172.21.15.176:0/3847628679 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f473800d600 con 0x7f4748072520 2024-02-19T22:08:22.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.906+0000 7f4737fff700 1 -- 172.21.15.176:0/3847628679 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f473800dbf0 con 0x7f4748072520 2024-02-19T22:08:22.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.906+0000 7f4737fff700 1 -- 172.21.15.176:0/3847628679 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4738011cb0 con 0x7f4748072520 2024-02-19T22:08:22.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.907+0000 7f474f786700 1 -- 172.21.15.176:0/3847628679 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4748072520 msgr2=0x7f4748070ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:22.910 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.907+0000 7f474f786700 1 --2- 172.21.15.176:0/3847628679 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4748072520 0x7f4748070ac0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f473800b5f0 tx=0x7f4738006ee0).stop 2024-02-19T22:08:22.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.907+0000 7f474f786700 1 -- 172.21.15.176:0/3847628679 shutdown_connections 2024-02-19T22:08:22.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.907+0000 7f474f786700 1 --2- 172.21.15.176:0/3847628679 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4748072520 0x7f4748070ac0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:22.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.907+0000 7f474f786700 1 -- 172.21.15.176:0/3847628679 >> 172.21.15.176:0/3847628679 conn(0x7f47480f76a0 msgr2=0x7f47480f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:22.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.907+0000 7f474f786700 1 -- 172.21.15.176:0/3847628679 shutdown_connections 2024-02-19T22:08:22.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.907+0000 7f474f786700 1 -- 172.21.15.176:0/3847628679 wait complete. 2024-02-19T22:08:22.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.907+0000 7f474f786700 1 Processor -- start 2024-02-19T22:08:22.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.907+0000 7f474f786700 1 -- start start 2024-02-19T22:08:22.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.907+0000 7f474f786700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4748072520 0x7f474810ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:22.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.907+0000 7f474f786700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474810d2f0 con 0x7f4748072520 2024-02-19T22:08:22.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.908+0000 7f474d522700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4748072520 0x7f474810ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:22.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.908+0000 7f474d522700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4748072520 0x7f474810ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38266/0 (socket says 172.21.15.176:38266) 2024-02-19T22:08:22.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.908+0000 7f474d522700 1 -- 172.21.15.176:0/3373678912 learned_addr learned my addr 172.21.15.176:0/3373678912 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:22.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.908+0000 7f474d522700 1 -- 172.21.15.176:0/3373678912 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473800b200 con 0x7f4748072520 2024-02-19T22:08:22.911 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.908+0000 7f474d522700 1 --2- 172.21.15.176:0/3373678912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4748072520 0x7f474810ce00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f4738006800 tx=0x7f4738006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:22.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.908+0000 7f4735ffb700 1 -- 172.21.15.176:0/3373678912 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f473801b570 con 0x7f4748072520 2024-02-19T22:08:22.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.908+0000 7f4735ffb700 1 -- 172.21.15.176:0/3373678912 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f473801bb60 con 0x7f4748072520 2024-02-19T22:08:22.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.908+0000 7f474f786700 1 -- 172.21.15.176:0/3373678912 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474810d4f0 con 0x7f4748072520 2024-02-19T22:08:22.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.908+0000 7f4735ffb700 1 -- 172.21.15.176:0/3373678912 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f473801a9f0 con 0x7f4748072520 2024-02-19T22:08:22.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.908+0000 7f474f786700 1 -- 172.21.15.176:0/3373678912 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f474810e100 con 0x7f4748072520 2024-02-19T22:08:22.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.909+0000 7f4735ffb700 1 -- 172.21.15.176:0/3373678912 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f473800f070 con 0x7f4748072520 2024-02-19T22:08:22.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.909+0000 7f4735ffb700 1 --2- 172.21.15.176:0/3373678912 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f473002ff70 0x7f4730032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:22.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.909+0000 7f4735ffb700 1 -- 172.21.15.176:0/3373678912 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4738041560 con 0x7f4748072520 2024-02-19T22:08:22.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.909+0000 7f474f786700 1 -- 172.21.15.176:0/3373678912 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4748059100 con 0x7f4748072520 2024-02-19T22:08:22.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.909+0000 7f474cd21700 1 --2- 172.21.15.176:0/3373678912 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f473002ff70 0x7f4730032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:22.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.910+0000 7f474cd21700 1 --2- 172.21.15.176:0/3373678912 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f473002ff70 0x7f4730032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f47440096a0 tx=0x7f4744006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:22.914 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:22.914+0000 7f4735ffb700 1 -- 172.21.15.176:0/3373678912 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4738016070 con 0x7f4748072520 2024-02-19T22:08:23.089 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:23.089+0000 7f474f786700 1 -- 172.21.15.176:0/3373678912 --> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f4748102160 con 0x7f473002ff70 2024-02-19T22:08:23.093 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:23.093+0000 7f4735ffb700 1 -- 172.21.15.176:0/3373678912 <== mgr.14142 v2:172.21.15.176:6800/918424985 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f4748102160 con 0x7f473002ff70 2024-02-19T22:08:23.093 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-19T22:08:23.095 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:23.095+0000 7f474f786700 1 -- 172.21.15.176:0/3373678912 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f473002ff70 msgr2=0x7f4730032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:23.096 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:23.095+0000 7f474f786700 1 --2- 172.21.15.176:0/3373678912 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f473002ff70 0x7f4730032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f47440096a0 tx=0x7f4744006b40).stop 2024-02-19T22:08:23.096 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:23.095+0000 7f474f786700 1 -- 172.21.15.176:0/3373678912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4748072520 msgr2=0x7f474810ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:23.096 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:23.095+0000 7f474f786700 1 --2- 172.21.15.176:0/3373678912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4748072520 0x7f474810ce00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f4738006800 tx=0x7f4738006ee0).stop 2024-02-19T22:08:23.096 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:23.095+0000 7f474f786700 1 -- 172.21.15.176:0/3373678912 shutdown_connections 2024-02-19T22:08:23.096 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:23.095+0000 7f474f786700 1 --2- 172.21.15.176:0/3373678912 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f473002ff70 0x7f4730032430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:23.096 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:23.095+0000 7f474f786700 1 --2- 172.21.15.176:0/3373678912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4748072520 0x7f474810ce00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:23.096 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:23.095+0000 7f474f786700 1 -- 172.21.15.176:0/3373678912 >> 172.21.15.176:0/3373678912 conn(0x7f47480f76a0 msgr2=0x7f47480f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:23.096 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:23.095+0000 7f474f786700 1 -- 172.21.15.176:0/3373678912 shutdown_connections 2024-02-19T22:08:23.096 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:23.095+0000 7f474f786700 1 -- 172.21.15.176:0/3373678912 wait complete. 2024-02-19T22:08:23.118 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:22 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:21.721466+0000 mon.smithi176 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi176.faoogw(active, since 1.05351s) 2024-02-19T22:08:23.118 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:22 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:08:21.722032+0000 mgr.smithi176.faoogw (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-19T22:08:23.119 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:22 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:21.724034+0000 mon.smithi176 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:23.693 INFO:teuthology.orchestra.run.smithi176.stderr:Deploying node-exporter service with default placement... 2024-02-19T22:08:23.693 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-02-19T22:08:23.948 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:22.725222+0000 mon.smithi176 (mon.0) 78 : cluster [DBG] mgrmap e12: smithi176.faoogw(active, since 2s) 2024-02-19T22:08:23.948 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:23.093719+0000 mon.smithi176 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:24.236 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.235+0000 7fc4e2218700 1 Processor -- start 2024-02-19T22:08:24.236 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.235+0000 7fc4e2218700 1 -- start start 2024-02-19T22:08:24.236 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.235+0000 7fc4e2218700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4dc0704a0 0x7fc4dc0feb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:24.237 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.235+0000 7fc4e2218700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc0feff0 con 0x7fc4dc0704a0 2024-02-19T22:08:24.237 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.235+0000 7fc4db7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4dc0704a0 0x7fc4dc0feb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:24.237 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.235+0000 7fc4db7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4dc0704a0 0x7fc4dc0feb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38278/0 (socket says 172.21.15.176:38278) 2024-02-19T22:08:24.237 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.235+0000 7fc4db7fe700 1 -- 172.21.15.176:0/2424776635 learned_addr learned my addr 172.21.15.176:0/2424776635 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:24.237 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.236+0000 7fc4db7fe700 1 -- 172.21.15.176:0/2424776635 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc0ff130 con 0x7fc4dc0704a0 2024-02-19T22:08:24.237 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.236+0000 7fc4db7fe700 1 --2- 172.21.15.176:0/2424776635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4dc0704a0 0x7fc4dc0feb00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fc4cc00b5f0 tx=0x7fc4cc006ee0).ready entity=mon.0 client_cookie=63c9413c54903df1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:24.237 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.236+0000 7fc4da7fc700 1 -- 172.21.15.176:0/2424776635 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4cc00d600 con 0x7fc4dc0704a0 2024-02-19T22:08:24.237 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.236+0000 7fc4da7fc700 1 -- 172.21.15.176:0/2424776635 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc4cc00dbf0 con 0x7fc4dc0704a0 2024-02-19T22:08:24.237 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.236+0000 7fc4da7fc700 1 -- 172.21.15.176:0/2424776635 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4cc011cb0 con 0x7fc4dc0704a0 2024-02-19T22:08:24.239 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.237+0000 7fc4e2218700 1 -- 172.21.15.176:0/2424776635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4dc0704a0 msgr2=0x7fc4dc0feb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:24.239 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.237+0000 7fc4e2218700 1 --2- 172.21.15.176:0/2424776635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4dc0704a0 0x7fc4dc0feb00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fc4cc00b5f0 tx=0x7fc4cc006ee0).stop 2024-02-19T22:08:24.239 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.237+0000 7fc4e2218700 1 -- 172.21.15.176:0/2424776635 shutdown_connections 2024-02-19T22:08:24.239 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.237+0000 7fc4e2218700 1 --2- 172.21.15.176:0/2424776635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4dc0704a0 0x7fc4dc0feb00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:24.239 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.237+0000 7fc4e2218700 1 -- 172.21.15.176:0/2424776635 >> 172.21.15.176:0/2424776635 conn(0x7fc4dc0f65c0 msgr2=0x7fc4dc0f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:24.239 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.237+0000 7fc4e2218700 1 -- 172.21.15.176:0/2424776635 shutdown_connections 2024-02-19T22:08:24.239 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.237+0000 7fc4e2218700 1 -- 172.21.15.176:0/2424776635 wait complete. 2024-02-19T22:08:24.239 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.237+0000 7fc4e2218700 1 Processor -- start 2024-02-19T22:08:24.239 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.237+0000 7fc4e2218700 1 -- start start 2024-02-19T22:08:24.240 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.238+0000 7fc4e2218700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4dc0704a0 0x7fc4dc10abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:24.240 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.238+0000 7fc4e2218700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc10b0b0 con 0x7fc4dc0704a0 2024-02-19T22:08:24.240 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.238+0000 7fc4db7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4dc0704a0 0x7fc4dc10abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:24.240 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.238+0000 7fc4db7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4dc0704a0 0x7fc4dc10abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:38290/0 (socket says 172.21.15.176:38290) 2024-02-19T22:08:24.240 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.238+0000 7fc4db7fe700 1 -- 172.21.15.176:0/62175755 learned_addr learned my addr 172.21.15.176:0/62175755 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:24.240 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.238+0000 7fc4db7fe700 1 -- 172.21.15.176:0/62175755 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4cc00b200 con 0x7fc4dc0704a0 2024-02-19T22:08:24.240 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.238+0000 7fc4db7fe700 1 --2- 172.21.15.176:0/62175755 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4dc0704a0 0x7fc4dc10abc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fc4cc00dfc0 tx=0x7fc4cc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:24.240 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.238+0000 7fc4c3fff700 1 -- 172.21.15.176:0/62175755 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4cc01b410 con 0x7fc4dc0704a0 2024-02-19T22:08:24.240 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.238+0000 7fc4c3fff700 1 -- 172.21.15.176:0/62175755 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc4cc01ba00 con 0x7fc4dc0704a0 2024-02-19T22:08:24.240 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.238+0000 7fc4c3fff700 1 -- 172.21.15.176:0/62175755 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4cc01a990 con 0x7fc4dc0704a0 2024-02-19T22:08:24.240 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.238+0000 7fc4e2218700 1 -- 172.21.15.176:0/62175755 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4dc10b2b0 con 0x7fc4dc0704a0 2024-02-19T22:08:24.240 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.238+0000 7fc4e2218700 1 -- 172.21.15.176:0/62175755 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc10b740 con 0x7fc4dc0704a0 2024-02-19T22:08:24.240 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.239+0000 7fc4e2218700 1 -- 172.21.15.176:0/62175755 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4dc045ad0 con 0x7fc4dc0704a0 2024-02-19T22:08:24.241 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.239+0000 7fc4c3fff700 1 -- 172.21.15.176:0/62175755 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fc4cc00f070 con 0x7fc4dc0704a0 2024-02-19T22:08:24.241 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.240+0000 7fc4c3fff700 1 --2- 172.21.15.176:0/62175755 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fc4c802ff20 0x7fc4c80323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:24.241 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.240+0000 7fc4c3fff700 1 -- 172.21.15.176:0/62175755 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc4cc0414d0 con 0x7fc4dc0704a0 2024-02-19T22:08:24.241 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.240+0000 7fc4daffd700 1 --2- 172.21.15.176:0/62175755 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fc4c802ff20 0x7fc4c80323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:24.241 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.240+0000 7fc4daffd700 1 --2- 172.21.15.176:0/62175755 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fc4c802ff20 0x7fc4c80323e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fc4d00096a0 tx=0x7fc4d0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:24.243 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.244+0000 7fc4c3fff700 1 -- 172.21.15.176:0/62175755 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4cc022310 con 0x7fc4dc0704a0 2024-02-19T22:08:24.419 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.419+0000 7fc4e2218700 1 -- 172.21.15.176:0/62175755 --> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4dc10bfa0 con 0x7fc4c802ff20 2024-02-19T22:08:24.423 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.423+0000 7fc4c3fff700 1 -- 172.21.15.176:0/62175755 <== mgr.14142 v2:172.21.15.176:6800/918424985 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fc4dc10bfa0 con 0x7fc4c802ff20 2024-02-19T22:08:24.423 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-19T22:08:24.426 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.425+0000 7fc4e2218700 1 -- 172.21.15.176:0/62175755 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fc4c802ff20 msgr2=0x7fc4c80323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:24.426 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.425+0000 7fc4e2218700 1 --2- 172.21.15.176:0/62175755 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fc4c802ff20 0x7fc4c80323e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fc4d00096a0 tx=0x7fc4d0006b40).stop 2024-02-19T22:08:24.426 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.425+0000 7fc4e2218700 1 -- 172.21.15.176:0/62175755 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4dc0704a0 msgr2=0x7fc4dc10abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:24.426 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.425+0000 7fc4e2218700 1 --2- 172.21.15.176:0/62175755 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4dc0704a0 0x7fc4dc10abc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fc4cc00dfc0 tx=0x7fc4cc006ee0).stop 2024-02-19T22:08:24.426 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.425+0000 7fc4e2218700 1 -- 172.21.15.176:0/62175755 shutdown_connections 2024-02-19T22:08:24.426 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.425+0000 7fc4e2218700 1 --2- 172.21.15.176:0/62175755 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fc4c802ff20 0x7fc4c80323e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:24.426 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.425+0000 7fc4e2218700 1 --2- 172.21.15.176:0/62175755 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc4dc0704a0 0x7fc4dc10abc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:24.426 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.425+0000 7fc4e2218700 1 -- 172.21.15.176:0/62175755 >> 172.21.15.176:0/62175755 conn(0x7fc4dc0f65c0 msgr2=0x7fc4dc0f7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:24.426 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.425+0000 7fc4e2218700 1 -- 172.21.15.176:0/62175755 shutdown_connections 2024-02-19T22:08:24.426 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:24.425+0000 7fc4e2218700 1 -- 172.21.15.176:0/62175755 wait complete. 2024-02-19T22:08:24.974 INFO:teuthology.orchestra.run.smithi176.stderr:Deploying alertmanager service with default placement... 2024-02-19T22:08:24.974 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-02-19T22:08:24.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:24 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:23.091065+0000 mgr.smithi176.faoogw (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:08:24.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:24 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:08:23.091558+0000 mgr.smithi176.faoogw (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-19T22:08:24.977 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:24 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:24.423825+0000 mon.smithi176 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:25.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.540+0000 7fd49691d700 1 Processor -- start 2024-02-19T22:08:25.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.540+0000 7fd49691d700 1 -- start start 2024-02-19T22:08:25.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.541+0000 7fd49691d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd490070430 0x7fd4900fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:25.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.541+0000 7fd49691d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4900fef40 con 0x7fd490070430 2024-02-19T22:08:25.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.541+0000 7fd48ffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd490070430 0x7fd4900fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:25.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.541+0000 7fd48ffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd490070430 0x7fd4900fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40740/0 (socket says 172.21.15.176:40740) 2024-02-19T22:08:25.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.541+0000 7fd48ffff700 1 -- 172.21.15.176:0/1775877240 learned_addr learned my addr 172.21.15.176:0/1775877240 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:25.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.541+0000 7fd48ffff700 1 -- 172.21.15.176:0/1775877240 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4900ff080 con 0x7fd490070430 2024-02-19T22:08:25.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.541+0000 7fd48ffff700 1 --2- 172.21.15.176:0/1775877240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd490070430 0x7fd4900fea50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fd48000b5f0 tx=0x7fd480006ee0).ready entity=mon.0 client_cookie=167651a889fd2ce6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:25.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.541+0000 7fd48effd700 1 -- 172.21.15.176:0/1775877240 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd48000d600 con 0x7fd490070430 2024-02-19T22:08:25.542 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.541+0000 7fd48effd700 1 -- 172.21.15.176:0/1775877240 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd48000dbf0 con 0x7fd490070430 2024-02-19T22:08:25.545 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.541+0000 7fd48effd700 1 -- 172.21.15.176:0/1775877240 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd480011cb0 con 0x7fd490070430 2024-02-19T22:08:25.545 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.542+0000 7fd49691d700 1 -- 172.21.15.176:0/1775877240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd490070430 msgr2=0x7fd4900fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:25.545 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.542+0000 7fd49691d700 1 --2- 172.21.15.176:0/1775877240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd490070430 0x7fd4900fea50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fd48000b5f0 tx=0x7fd480006ee0).stop 2024-02-19T22:08:25.545 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.542+0000 7fd49691d700 1 -- 172.21.15.176:0/1775877240 shutdown_connections 2024-02-19T22:08:25.545 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.542+0000 7fd49691d700 1 --2- 172.21.15.176:0/1775877240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd490070430 0x7fd4900fea50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:25.545 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.542+0000 7fd49691d700 1 -- 172.21.15.176:0/1775877240 >> 172.21.15.176:0/1775877240 conn(0x7fd4900f6530 msgr2=0x7fd4900f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:25.545 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.542+0000 7fd49691d700 1 -- 172.21.15.176:0/1775877240 shutdown_connections 2024-02-19T22:08:25.545 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.542+0000 7fd49691d700 1 -- 172.21.15.176:0/1775877240 wait complete. 2024-02-19T22:08:25.545 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.542+0000 7fd49691d700 1 Processor -- start 2024-02-19T22:08:25.545 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.543+0000 7fd49691d700 1 -- start start 2024-02-19T22:08:25.545 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.543+0000 7fd49691d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd490070430 0x7fd49010ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:25.545 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.543+0000 7fd49691d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49010b030 con 0x7fd490070430 2024-02-19T22:08:25.545 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.543+0000 7fd48ffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd490070430 0x7fd49010ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:25.546 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.543+0000 7fd48ffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd490070430 0x7fd49010ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40756/0 (socket says 172.21.15.176:40756) 2024-02-19T22:08:25.546 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.543+0000 7fd48ffff700 1 -- 172.21.15.176:0/1235650113 learned_addr learned my addr 172.21.15.176:0/1235650113 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:25.546 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.543+0000 7fd48ffff700 1 -- 172.21.15.176:0/1235650113 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48000b200 con 0x7fd490070430 2024-02-19T22:08:25.546 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.543+0000 7fd48ffff700 1 --2- 172.21.15.176:0/1235650113 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd490070430 0x7fd49010ab40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fd480006800 tx=0x7fd480006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:25.546 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.543+0000 7fd48cff9700 1 -- 172.21.15.176:0/1235650113 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd48001b570 con 0x7fd490070430 2024-02-19T22:08:25.546 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.543+0000 7fd49691d700 1 -- 172.21.15.176:0/1235650113 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49010b230 con 0x7fd490070430 2024-02-19T22:08:25.546 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.543+0000 7fd48cff9700 1 -- 172.21.15.176:0/1235650113 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd48001bb60 con 0x7fd490070430 2024-02-19T22:08:25.546 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.544+0000 7fd48cff9700 1 -- 172.21.15.176:0/1235650113 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd48001a9f0 con 0x7fd490070430 2024-02-19T22:08:25.546 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.544+0000 7fd49691d700 1 -- 172.21.15.176:0/1235650113 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49010b6c0 con 0x7fd490070430 2024-02-19T22:08:25.546 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.544+0000 7fd49691d700 1 -- 172.21.15.176:0/1235650113 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd490045ad0 con 0x7fd490070430 2024-02-19T22:08:25.546 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.544+0000 7fd48cff9700 1 -- 172.21.15.176:0/1235650113 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fd48000f070 con 0x7fd490070430 2024-02-19T22:08:25.546 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.545+0000 7fd48cff9700 1 --2- 172.21.15.176:0/1235650113 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fd47002ff20 0x7fd4700323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:25.546 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.545+0000 7fd48cff9700 1 -- 172.21.15.176:0/1235650113 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd480041620 con 0x7fd490070430 2024-02-19T22:08:25.546 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.545+0000 7fd48f7fe700 1 --2- 172.21.15.176:0/1235650113 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fd47002ff20 0x7fd4700323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:25.547 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.545+0000 7fd48f7fe700 1 --2- 172.21.15.176:0/1235650113 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fd47002ff20 0x7fd4700323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd4780096a0 tx=0x7fd478006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:25.549 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.549+0000 7fd48cff9700 1 -- 172.21.15.176:0/1235650113 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd480022310 con 0x7fd490070430 2024-02-19T22:08:25.723 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.723+0000 7fd49691d700 1 -- 172.21.15.176:0/1235650113 --> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fd49010bf20 con 0x7fd47002ff20 2024-02-19T22:08:25.728 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.728+0000 7fd48cff9700 1 -- 172.21.15.176:0/1235650113 <== mgr.14142 v2:172.21.15.176:6800/918424985 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fd49010bf20 con 0x7fd47002ff20 2024-02-19T22:08:25.728 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-19T22:08:25.730 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.729+0000 7fd49691d700 1 -- 172.21.15.176:0/1235650113 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fd47002ff20 msgr2=0x7fd4700323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:25.730 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.729+0000 7fd49691d700 1 --2- 172.21.15.176:0/1235650113 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fd47002ff20 0x7fd4700323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd4780096a0 tx=0x7fd478006b40).stop 2024-02-19T22:08:25.730 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.729+0000 7fd49691d700 1 -- 172.21.15.176:0/1235650113 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd490070430 msgr2=0x7fd49010ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:25.730 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.729+0000 7fd49691d700 1 --2- 172.21.15.176:0/1235650113 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd490070430 0x7fd49010ab40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fd480006800 tx=0x7fd480006ee0).stop 2024-02-19T22:08:25.730 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.730+0000 7fd49691d700 1 -- 172.21.15.176:0/1235650113 shutdown_connections 2024-02-19T22:08:25.731 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.730+0000 7fd49691d700 1 --2- 172.21.15.176:0/1235650113 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fd47002ff20 0x7fd4700323e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:25.731 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.730+0000 7fd49691d700 1 --2- 172.21.15.176:0/1235650113 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd490070430 0x7fd49010ab40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:25.731 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.730+0000 7fd49691d700 1 -- 172.21.15.176:0/1235650113 >> 172.21.15.176:0/1235650113 conn(0x7fd4900f6530 msgr2=0x7fd4900f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:25.731 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.730+0000 7fd49691d700 1 -- 172.21.15.176:0/1235650113 shutdown_connections 2024-02-19T22:08:25.731 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:25.730+0000 7fd49691d700 1 -- 172.21.15.176:0/1235650113 wait complete. 2024-02-19T22:08:26.036 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:24.420943+0000 mgr.smithi176.faoogw (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:08:26.036 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:08:24.421582+0000 mgr.smithi176.faoogw (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-19T22:08:26.272 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-02-19T22:08:26.828 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.824+0000 7ff27d41d700 1 Processor -- start 2024-02-19T22:08:26.828 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.824+0000 7ff27d41d700 1 -- start start 2024-02-19T22:08:26.828 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.825+0000 7ff27d41d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2780fbd90 0x7ff2780fc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:26.828 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.825+0000 7ff27d41d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2780fc6a0 con 0x7ff2780fbd90 2024-02-19T22:08:26.828 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.825+0000 7ff277fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2780fbd90 0x7ff2780fc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:26.828 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.825+0000 7ff277fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2780fbd90 0x7ff2780fc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40766/0 (socket says 172.21.15.176:40766) 2024-02-19T22:08:26.828 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.825+0000 7ff277fff700 1 -- 172.21.15.176:0/3179667119 learned_addr learned my addr 172.21.15.176:0/3179667119 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:26.828 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.825+0000 7ff277fff700 1 -- 172.21.15.176:0/3179667119 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2780fc7e0 con 0x7ff2780fbd90 2024-02-19T22:08:26.828 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.825+0000 7ff277fff700 1 --2- 172.21.15.176:0/3179667119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2780fbd90 0x7ff2780fc1b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ff26800b5f0 tx=0x7ff268006ee0).ready entity=mon.0 client_cookie=96224e9ff428ffcb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:26.828 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.825+0000 7ff276ffd700 1 -- 172.21.15.176:0/3179667119 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff26800d600 con 0x7ff2780fbd90 2024-02-19T22:08:26.828 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.825+0000 7ff276ffd700 1 -- 172.21.15.176:0/3179667119 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff26800dbf0 con 0x7ff2780fbd90 2024-02-19T22:08:26.828 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.825+0000 7ff276ffd700 1 -- 172.21.15.176:0/3179667119 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff268011cb0 con 0x7ff2780fbd90 2024-02-19T22:08:26.829 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.826+0000 7ff27d41d700 1 -- 172.21.15.176:0/3179667119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2780fbd90 msgr2=0x7ff2780fc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:26.829 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.826+0000 7ff27d41d700 1 --2- 172.21.15.176:0/3179667119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2780fbd90 0x7ff2780fc1b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ff26800b5f0 tx=0x7ff268006ee0).stop 2024-02-19T22:08:26.829 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.826+0000 7ff27d41d700 1 -- 172.21.15.176:0/3179667119 shutdown_connections 2024-02-19T22:08:26.829 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.826+0000 7ff27d41d700 1 --2- 172.21.15.176:0/3179667119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2780fbd90 0x7ff2780fc1b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:26.829 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.826+0000 7ff27d41d700 1 -- 172.21.15.176:0/3179667119 >> 172.21.15.176:0/3179667119 conn(0x7ff2780f7510 msgr2=0x7ff2780f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:26.829 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.826+0000 7ff27d41d700 1 -- 172.21.15.176:0/3179667119 shutdown_connections 2024-02-19T22:08:26.829 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.826+0000 7ff27d41d700 1 -- 172.21.15.176:0/3179667119 wait complete. 2024-02-19T22:08:26.829 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.826+0000 7ff27d41d700 1 Processor -- start 2024-02-19T22:08:26.829 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.827+0000 7ff27d41d700 1 -- start start 2024-02-19T22:08:26.829 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.827+0000 7ff27d41d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2780fbd90 0x7ff27810a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:26.829 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.827+0000 7ff27d41d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27810ae30 con 0x7ff2780fbd90 2024-02-19T22:08:26.829 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.827+0000 7ff277fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2780fbd90 0x7ff27810a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:26.830 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.827+0000 7ff277fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2780fbd90 0x7ff27810a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40772/0 (socket says 172.21.15.176:40772) 2024-02-19T22:08:26.830 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.827+0000 7ff277fff700 1 -- 172.21.15.176:0/1793713425 learned_addr learned my addr 172.21.15.176:0/1793713425 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:26.830 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.827+0000 7ff277fff700 1 -- 172.21.15.176:0/1793713425 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26800b200 con 0x7ff2780fbd90 2024-02-19T22:08:26.830 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.827+0000 7ff277fff700 1 --2- 172.21.15.176:0/1793713425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2780fbd90 0x7ff27810a940 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff268011ea0 tx=0x7ff268006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:26.830 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.827+0000 7ff274ff9700 1 -- 172.21.15.176:0/1793713425 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff26801b570 con 0x7ff2780fbd90 2024-02-19T22:08:26.830 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.827+0000 7ff274ff9700 1 -- 172.21.15.176:0/1793713425 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff26801bb60 con 0x7ff2780fbd90 2024-02-19T22:08:26.830 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.827+0000 7ff274ff9700 1 -- 172.21.15.176:0/1793713425 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff26801a8d0 con 0x7ff2780fbd90 2024-02-19T22:08:26.830 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.828+0000 7ff27d41d700 1 -- 172.21.15.176:0/1793713425 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff27810b030 con 0x7ff2780fbd90 2024-02-19T22:08:26.830 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.828+0000 7ff27d41d700 1 -- 172.21.15.176:0/1793713425 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff27810b4c0 con 0x7ff2780fbd90 2024-02-19T22:08:26.831 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.828+0000 7ff274ff9700 1 -- 172.21.15.176:0/1793713425 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ff26800f070 con 0x7ff2780fbd90 2024-02-19T22:08:26.831 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.828+0000 7ff27d41d700 1 -- 172.21.15.176:0/1793713425 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff278045ad0 con 0x7ff2780fbd90 2024-02-19T22:08:26.831 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.829+0000 7ff274ff9700 1 --2- 172.21.15.176:0/1793713425 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7ff26402ff70 0x7ff264032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:26.831 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.829+0000 7ff274ff9700 1 -- 172.21.15.176:0/1793713425 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff2680414e0 con 0x7ff2780fbd90 2024-02-19T22:08:26.831 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.829+0000 7ff2777fe700 1 --2- 172.21.15.176:0/1793713425 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7ff26402ff70 0x7ff264032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:26.831 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.829+0000 7ff2777fe700 1 --2- 172.21.15.176:0/1793713425 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7ff26402ff70 0x7ff264032430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7ff26c0096a0 tx=0x7ff26c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:26.832 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.833+0000 7ff274ff9700 1 -- 172.21.15.176:0/1793713425 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff268019ab0 con 0x7ff2780fbd90 2024-02-19T22:08:26.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:26 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:25.724547+0000 mgr.smithi176.faoogw (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:08:26.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:26 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:08:25.725055+0000 mgr.smithi176.faoogw (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-19T22:08:26.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:26 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:25.728367+0000 mon.smithi176 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:26.981 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.981+0000 7ff27d41d700 1 -- 172.21.15.176:0/1793713425 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7ff2780fda70 con 0x7ff2780fbd90 2024-02-19T22:08:26.984 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.984+0000 7ff274ff9700 1 -- 172.21.15.176:0/1793713425 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7ff268016070 con 0x7ff2780fbd90 2024-02-19T22:08:26.987 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.986+0000 7ff27d41d700 1 -- 172.21.15.176:0/1793713425 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7ff26402ff70 msgr2=0x7ff264032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:26.987 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.986+0000 7ff27d41d700 1 --2- 172.21.15.176:0/1793713425 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7ff26402ff70 0x7ff264032430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7ff26c0096a0 tx=0x7ff26c006b40).stop 2024-02-19T22:08:26.987 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.986+0000 7ff27d41d700 1 -- 172.21.15.176:0/1793713425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2780fbd90 msgr2=0x7ff27810a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:26.987 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.986+0000 7ff27d41d700 1 --2- 172.21.15.176:0/1793713425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2780fbd90 0x7ff27810a940 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff268011ea0 tx=0x7ff268006ee0).stop 2024-02-19T22:08:26.987 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.986+0000 7ff27d41d700 1 -- 172.21.15.176:0/1793713425 shutdown_connections 2024-02-19T22:08:26.988 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.986+0000 7ff27d41d700 1 --2- 172.21.15.176:0/1793713425 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7ff26402ff70 0x7ff264032430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:26.988 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.986+0000 7ff27d41d700 1 --2- 172.21.15.176:0/1793713425 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2780fbd90 0x7ff27810a940 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:26.988 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.986+0000 7ff27d41d700 1 -- 172.21.15.176:0/1793713425 >> 172.21.15.176:0/1793713425 conn(0x7ff2780f7510 msgr2=0x7ff2780f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:26.988 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.986+0000 7ff27d41d700 1 -- 172.21.15.176:0/1793713425 shutdown_connections 2024-02-19T22:08:26.988 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:26.986+0000 7ff27d41d700 1 -- 172.21.15.176:0/1793713425 wait complete. 2024-02-19T22:08:27.562 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-02-19T22:08:28.075 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.068+0000 7f7cb8c7e700 1 Processor -- start 2024-02-19T22:08:28.075 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.068+0000 7f7cb8c7e700 1 -- start start 2024-02-19T22:08:28.075 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.068+0000 7f7cb8c7e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb4070430 0x7f7cb40fe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:28.075 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.068+0000 7f7cb8c7e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb40fed80 con 0x7f7cb4070430 2024-02-19T22:08:28.075 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.069+0000 7f7cb37fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb4070430 0x7f7cb40fe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:28.075 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.069+0000 7f7cb37fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb4070430 0x7f7cb40fe890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40778/0 (socket says 172.21.15.176:40778) 2024-02-19T22:08:28.075 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.069+0000 7f7cb37fe700 1 -- 172.21.15.176:0/3290362453 learned_addr learned my addr 172.21.15.176:0/3290362453 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:28.075 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.069+0000 7f7cb37fe700 1 -- 172.21.15.176:0/3290362453 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb40feec0 con 0x7f7cb4070430 2024-02-19T22:08:28.075 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.069+0000 7f7cb37fe700 1 --2- 172.21.15.176:0/3290362453 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb4070430 0x7f7cb40fe890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7ca400b5f0 tx=0x7f7ca4006ee0).ready entity=mon.0 client_cookie=1e9952814d8d92d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:28.075 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.069+0000 7f7cb27fc700 1 -- 172.21.15.176:0/3290362453 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ca400d600 con 0x7f7cb4070430 2024-02-19T22:08:28.076 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.069+0000 7f7cb27fc700 1 -- 172.21.15.176:0/3290362453 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7ca400dbf0 con 0x7f7cb4070430 2024-02-19T22:08:28.076 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.069+0000 7f7cb27fc700 1 -- 172.21.15.176:0/3290362453 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ca4011cb0 con 0x7f7cb4070430 2024-02-19T22:08:28.076 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.070+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/3290362453 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb4070430 msgr2=0x7f7cb40fe890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:28.076 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.070+0000 7f7cb8c7e700 1 --2- 172.21.15.176:0/3290362453 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb4070430 0x7f7cb40fe890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7ca400b5f0 tx=0x7f7ca4006ee0).stop 2024-02-19T22:08:28.076 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.070+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/3290362453 shutdown_connections 2024-02-19T22:08:28.076 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.070+0000 7f7cb8c7e700 1 --2- 172.21.15.176:0/3290362453 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb4070430 0x7f7cb40fe890 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:28.076 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.070+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/3290362453 >> 172.21.15.176:0/3290362453 conn(0x7f7cb40f6390 msgr2=0x7f7cb40f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:28.076 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.070+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/3290362453 shutdown_connections 2024-02-19T22:08:28.076 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.070+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/3290362453 wait complete. 2024-02-19T22:08:28.076 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.070+0000 7f7cb8c7e700 1 Processor -- start 2024-02-19T22:08:28.076 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.070+0000 7f7cb8c7e700 1 -- start start 2024-02-19T22:08:28.076 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.071+0000 7f7cb8c7e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb4070430 0x7f7cb40fc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:28.076 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.071+0000 7f7cb8c7e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb40fa790 con 0x7f7cb4070430 2024-02-19T22:08:28.077 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.071+0000 7f7cb37fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb4070430 0x7f7cb40fc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:28.077 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.071+0000 7f7cb37fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb4070430 0x7f7cb40fc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40782/0 (socket says 172.21.15.176:40782) 2024-02-19T22:08:28.077 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.071+0000 7f7cb37fe700 1 -- 172.21.15.176:0/891385217 learned_addr learned my addr 172.21.15.176:0/891385217 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:28.077 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.071+0000 7f7cb37fe700 1 -- 172.21.15.176:0/891385217 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca400b200 con 0x7f7cb4070430 2024-02-19T22:08:28.077 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.071+0000 7f7cb37fe700 1 --2- 172.21.15.176:0/891385217 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb4070430 0x7f7cb40fc310 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f7ca4006800 tx=0x7f7ca4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:28.077 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.071+0000 7f7cabfff700 1 -- 172.21.15.176:0/891385217 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ca401b570 con 0x7f7cb4070430 2024-02-19T22:08:28.077 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.071+0000 7f7cabfff700 1 -- 172.21.15.176:0/891385217 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7ca401bb60 con 0x7f7cb4070430 2024-02-19T22:08:28.077 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.071+0000 7f7cabfff700 1 -- 172.21.15.176:0/891385217 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ca401a9f0 con 0x7f7cb4070430 2024-02-19T22:08:28.077 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.071+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/891385217 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb40fa930 con 0x7f7cb4070430 2024-02-19T22:08:28.077 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.071+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/891385217 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb40fadc0 con 0x7f7cb4070430 2024-02-19T22:08:28.077 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.072+0000 7f7cabfff700 1 -- 172.21.15.176:0/891385217 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f7ca400f070 con 0x7f7cb4070430 2024-02-19T22:08:28.077 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.072+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/891385217 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb4045ad0 con 0x7f7cb4070430 2024-02-19T22:08:28.077 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.072+0000 7f7cabfff700 1 --2- 172.21.15.176:0/891385217 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f7c9402ff20 0x7f7c940323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:28.077 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.073+0000 7f7cabfff700 1 -- 172.21.15.176:0/891385217 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7ca4041590 con 0x7f7cb4070430 2024-02-19T22:08:28.078 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.073+0000 7f7cb2ffd700 1 --2- 172.21.15.176:0/891385217 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f7c9402ff20 0x7f7c940323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:28.078 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.073+0000 7f7cb2ffd700 1 --2- 172.21.15.176:0/891385217 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f7c9402ff20 0x7f7c940323e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7c9c0096a0 tx=0x7f7c9c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:28.078 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.076+0000 7f7cabfff700 1 -- 172.21.15.176:0/891385217 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ca4019aa0 con 0x7f7cb4070430 2024-02-19T22:08:28.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:26.984758+0000 mon.smithi176 (mon.0) 82 : audit [INF] from='client.? 172.21.15.176:0/1793713425' entity='client.admin' 2024-02-19T22:08:28.228 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.228+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/891385217 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f7cb40ffc00 con 0x7f7cb4070430 2024-02-19T22:08:28.230 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.230+0000 7f7cabfff700 1 -- 172.21.15.176:0/891385217 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f7ca4016070 con 0x7f7cb4070430 2024-02-19T22:08:28.233 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.232+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/891385217 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f7c9402ff20 msgr2=0x7f7c940323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:28.233 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.232+0000 7f7cb8c7e700 1 --2- 172.21.15.176:0/891385217 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f7c9402ff20 0x7f7c940323e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7c9c0096a0 tx=0x7f7c9c006b40).stop 2024-02-19T22:08:28.233 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.232+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/891385217 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb4070430 msgr2=0x7f7cb40fc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:28.233 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.232+0000 7f7cb8c7e700 1 --2- 172.21.15.176:0/891385217 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb4070430 0x7f7cb40fc310 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f7ca4006800 tx=0x7f7ca4006ee0).stop 2024-02-19T22:08:28.233 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.232+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/891385217 shutdown_connections 2024-02-19T22:08:28.233 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.232+0000 7f7cb8c7e700 1 --2- 172.21.15.176:0/891385217 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f7c9402ff20 0x7f7c940323e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:28.233 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.232+0000 7f7cb8c7e700 1 --2- 172.21.15.176:0/891385217 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cb4070430 0x7f7cb40fc310 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:28.234 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.232+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/891385217 >> 172.21.15.176:0/891385217 conn(0x7f7cb40f6390 msgr2=0x7f7cb40f7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:28.234 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.233+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/891385217 shutdown_connections 2024-02-19T22:08:28.234 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:28.233+0000 7f7cb8c7e700 1 -- 172.21.15.176:0/891385217 wait complete. 2024-02-19T22:08:28.896 INFO:teuthology.orchestra.run.smithi176.stderr:Enabling the dashboard module... 2024-02-19T22:08:28.896 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-02-19T22:08:29.384 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:29 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:28.231313+0000 mon.smithi176 (mon.0) 83 : audit [INF] from='client.? 172.21.15.176:0/891385217' entity='client.admin' 2024-02-19T22:08:29.937 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.618+0000 7f6b9d7d5700 1 Processor -- start 2024-02-19T22:08:29.937 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.618+0000 7f6b9d7d5700 1 -- start start 2024-02-19T22:08:29.937 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.618+0000 7f6b9d7d5700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b980fc4e0 0x7f6b980fc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:29.937 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.618+0000 7f6b9d7d5700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b980fcdf0 con 0x7f6b980fc4e0 2024-02-19T22:08:29.938 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.619+0000 7f6b96ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b980fc4e0 0x7f6b980fc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:29.938 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.619+0000 7f6b96ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b980fc4e0 0x7f6b980fc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40788/0 (socket says 172.21.15.176:40788) 2024-02-19T22:08:29.938 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.619+0000 7f6b96ffd700 1 -- 172.21.15.176:0/99677981 learned_addr learned my addr 172.21.15.176:0/99677981 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:29.938 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.619+0000 7f6b96ffd700 1 -- 172.21.15.176:0/99677981 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b980fcf30 con 0x7f6b980fc4e0 2024-02-19T22:08:29.938 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.619+0000 7f6b96ffd700 1 --2- 172.21.15.176:0/99677981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b980fc4e0 0x7f6b980fc900 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f6b9000b5f0 tx=0x7f6b90006ee0).ready entity=mon.0 client_cookie=d7bcb1d4eb8ed45a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:29.938 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.619+0000 7f6b95ffb700 1 -- 172.21.15.176:0/99677981 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b9000d600 con 0x7f6b980fc4e0 2024-02-19T22:08:29.938 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.619+0000 7f6b95ffb700 1 -- 172.21.15.176:0/99677981 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6b9000dbf0 con 0x7f6b980fc4e0 2024-02-19T22:08:29.938 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.619+0000 7f6b95ffb700 1 -- 172.21.15.176:0/99677981 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b90011cb0 con 0x7f6b980fc4e0 2024-02-19T22:08:29.938 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.620+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/99677981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b980fc4e0 msgr2=0x7f6b980fc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:29.938 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.620+0000 7f6b9d7d5700 1 --2- 172.21.15.176:0/99677981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b980fc4e0 0x7f6b980fc900 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f6b9000b5f0 tx=0x7f6b90006ee0).stop 2024-02-19T22:08:29.938 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.620+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/99677981 shutdown_connections 2024-02-19T22:08:29.938 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.620+0000 7f6b9d7d5700 1 --2- 172.21.15.176:0/99677981 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b980fc4e0 0x7f6b980fc900 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:29.938 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.620+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/99677981 >> 172.21.15.176:0/99677981 conn(0x7f6b980f5d30 msgr2=0x7f6b980f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:29.938 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.620+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/99677981 shutdown_connections 2024-02-19T22:08:29.939 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.620+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/99677981 wait complete. 2024-02-19T22:08:29.939 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.620+0000 7f6b9d7d5700 1 Processor -- start 2024-02-19T22:08:29.939 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.620+0000 7f6b9d7d5700 1 -- start start 2024-02-19T22:08:29.939 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.620+0000 7f6b9d7d5700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b9810ce90 0x7f6b9810d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:29.939 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.620+0000 7f6b9d7d5700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9810d7a0 con 0x7f6b9810ce90 2024-02-19T22:08:29.939 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.621+0000 7f6b96ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b9810ce90 0x7f6b9810d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:29.939 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.621+0000 7f6b96ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b9810ce90 0x7f6b9810d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40804/0 (socket says 172.21.15.176:40804) 2024-02-19T22:08:29.939 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.621+0000 7f6b96ffd700 1 -- 172.21.15.176:0/796245899 learned_addr learned my addr 172.21.15.176:0/796245899 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:29.939 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.621+0000 7f6b96ffd700 1 -- 172.21.15.176:0/796245899 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9000b200 con 0x7f6b9810ce90 2024-02-19T22:08:29.939 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.621+0000 7f6b96ffd700 1 --2- 172.21.15.176:0/796245899 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b9810ce90 0x7f6b9810d2b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f6b9000df90 tx=0x7f6b90006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:29.939 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.621+0000 7f6b7f7fe700 1 -- 172.21.15.176:0/796245899 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b90011400 con 0x7f6b9810ce90 2024-02-19T22:08:29.939 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.621+0000 7f6b7f7fe700 1 -- 172.21.15.176:0/796245899 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6b9001b8a0 con 0x7f6b9810ce90 2024-02-19T22:08:29.939 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.621+0000 7f6b7f7fe700 1 -- 172.21.15.176:0/796245899 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b9001a780 con 0x7f6b9810ce90 2024-02-19T22:08:29.939 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.621+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/796245899 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9810d9a0 con 0x7f6b9810ce90 2024-02-19T22:08:29.940 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.621+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/796245899 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9810e4f0 con 0x7f6b9810ce90 2024-02-19T22:08:29.940 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.622+0000 7f6b7f7fe700 1 -- 172.21.15.176:0/796245899 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f6b9000f070 con 0x7f6b9810ce90 2024-02-19T22:08:29.940 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.622+0000 7f6b7f7fe700 1 --2- 172.21.15.176:0/796245899 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f6b8402ff20 0x7f6b840323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:29.940 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.622+0000 7f6b7f7fe700 1 -- 172.21.15.176:0/796245899 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f6b90041450 con 0x7f6b9810ce90 2024-02-19T22:08:29.940 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.622+0000 7f6b967fc700 1 --2- 172.21.15.176:0/796245899 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f6b8402ff20 0x7f6b840323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:29.940 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.622+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/796245899 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b98059100 con 0x7f6b9810ce90 2024-02-19T22:08:29.940 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.622+0000 7f6b967fc700 1 --2- 172.21.15.176:0/796245899 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f6b8402ff20 0x7f6b840323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f6b8c0096a0 tx=0x7f6b8c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:29.940 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.626+0000 7f6b7f7fe700 1 -- 172.21.15.176:0/796245899 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b90016070 con 0x7f6b9810ce90 2024-02-19T22:08:29.940 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:29.817+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/796245899 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f6b98106d50 con 0x7f6b9810ce90 2024-02-19T22:08:30.343 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:30.342+0000 7f6b7f7fe700 1 -- 172.21.15.176:0/796245899 <== mon.0 v2:172.21.15.176:3300/0 7 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b900228e0 con 0x7f6b9810ce90 2024-02-19T22:08:30.723 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:30.723+0000 7f6b7f7fe700 1 -- 172.21.15.176:0/796245899 <== mon.0 v2:172.21.15.176:3300/0 8 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f6b9001a8e0 con 0x7f6b9810ce90 2024-02-19T22:08:30.724 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:30.724+0000 7f6b7f7fe700 1 -- 172.21.15.176:0/796245899 <== mon.0 v2:172.21.15.176:3300/0 9 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f6b90019770 con 0x7f6b9810ce90 2024-02-19T22:08:30.727 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:30.726+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/796245899 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f6b8402ff20 msgr2=0x7f6b840323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:30.727 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:30.726+0000 7f6b9d7d5700 1 --2- 172.21.15.176:0/796245899 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f6b8402ff20 0x7f6b840323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f6b8c0096a0 tx=0x7f6b8c006b40).stop 2024-02-19T22:08:30.727 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:30.726+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/796245899 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b9810ce90 msgr2=0x7f6b9810d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:30.728 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:30.726+0000 7f6b9d7d5700 1 --2- 172.21.15.176:0/796245899 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b9810ce90 0x7f6b9810d2b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f6b9000df90 tx=0x7f6b90006ee0).stop 2024-02-19T22:08:30.728 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:30.727+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/796245899 shutdown_connections 2024-02-19T22:08:30.728 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:30.727+0000 7f6b9d7d5700 1 --2- 172.21.15.176:0/796245899 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f6b8402ff20 0x7f6b840323e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:30.728 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:30.727+0000 7f6b9d7d5700 1 --2- 172.21.15.176:0/796245899 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6b9810ce90 0x7f6b9810d2b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:30.728 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:30.727+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/796245899 >> 172.21.15.176:0/796245899 conn(0x7f6b980f5d30 msgr2=0x7f6b980f7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:30.728 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:30.728+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/796245899 shutdown_connections 2024-02-19T22:08:30.728 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:30.728+0000 7f6b9d7d5700 1 -- 172.21.15.176:0/796245899 wait complete. 2024-02-19T22:08:31.038 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:30 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:29.818554+0000 mon.smithi176 (mon.0) 84 : audit [INF] from='client.? 172.21.15.176:0/796245899' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-19T22:08:31.039 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:30 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:30.343496+0000 mon.smithi176 (mon.0) 85 : audit [INF] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:31.039 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:30 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:30.557678+0000 mon.smithi176 (mon.0) 86 : audit [INF] from='mgr.14142 172.21.15.176:0/3240405657' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:31.300 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-19T22:08:31.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:30.725106+0000 mon.smithi176 (mon.0) 87 : audit [INF] from='client.? 172.21.15.176:0/796245899' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-19T22:08:31.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:30.725148+0000 mon.smithi176 (mon.0) 88 : cluster [DBG] mgrmap e13: smithi176.faoogw(active, since 10s) 2024-02-19T22:08:32.057 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.006+0000 7f758d3c7700 1 Processor -- start 2024-02-19T22:08:32.057 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.006+0000 7f758d3c7700 1 -- start start 2024-02-19T22:08:32.057 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.006+0000 7f758d3c7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75880fafa0 0x7f75880fb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:32.057 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.006+0000 7f758d3c7700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75880fb8b0 con 0x7f75880fafa0 2024-02-19T22:08:32.057 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.006+0000 7f7586ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75880fafa0 0x7f75880fb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:32.057 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.006+0000 7f7586ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75880fafa0 0x7f75880fb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40830/0 (socket says 172.21.15.176:40830) 2024-02-19T22:08:32.057 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.006+0000 7f7586ffd700 1 -- 172.21.15.176:0/2910688446 learned_addr learned my addr 172.21.15.176:0/2910688446 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.007+0000 7f7586ffd700 1 -- 172.21.15.176:0/2910688446 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75880fb9f0 con 0x7f75880fafa0 2024-02-19T22:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.007+0000 7f7586ffd700 1 --2- 172.21.15.176:0/2910688446 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75880fafa0 0x7f75880fb3c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f757800b5f0 tx=0x7f7578006ee0).ready entity=mon.0 client_cookie=3ac54d88f3b6442f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.007+0000 7f7585ffb700 1 -- 172.21.15.176:0/2910688446 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f757800d600 con 0x7f75880fafa0 2024-02-19T22:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.007+0000 7f7585ffb700 1 -- 172.21.15.176:0/2910688446 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f757800dc30 con 0x7f75880fafa0 2024-02-19T22:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.007+0000 7f7585ffb700 1 -- 172.21.15.176:0/2910688446 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7578011d30 con 0x7f75880fafa0 2024-02-19T22:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.008+0000 7f758d3c7700 1 -- 172.21.15.176:0/2910688446 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75880fafa0 msgr2=0x7f75880fb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.008+0000 7f758d3c7700 1 --2- 172.21.15.176:0/2910688446 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75880fafa0 0x7f75880fb3c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f757800b5f0 tx=0x7f7578006ee0).stop 2024-02-19T22:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.008+0000 7f758d3c7700 1 -- 172.21.15.176:0/2910688446 shutdown_connections 2024-02-19T22:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.008+0000 7f758d3c7700 1 --2- 172.21.15.176:0/2910688446 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75880fafa0 0x7f75880fb3c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.008+0000 7f758d3c7700 1 -- 172.21.15.176:0/2910688446 >> 172.21.15.176:0/2910688446 conn(0x7f75880f5ae0 msgr2=0x7f75880f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.008+0000 7f758d3c7700 1 -- 172.21.15.176:0/2910688446 shutdown_connections 2024-02-19T22:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.008+0000 7f758d3c7700 1 -- 172.21.15.176:0/2910688446 wait complete. 2024-02-19T22:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.009+0000 7f758d3c7700 1 Processor -- start 2024-02-19T22:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.009+0000 7f758d3c7700 1 -- start start 2024-02-19T22:08:32.059 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.009+0000 7f758d3c7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75880fafa0 0x7f7588068860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:32.059 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.009+0000 7f758d3c7700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7588068d50 con 0x7f75880fafa0 2024-02-19T22:08:32.059 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.009+0000 7f7586ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75880fafa0 0x7f7588068860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:32.059 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.009+0000 7f7586ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75880fafa0 0x7f7588068860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40838/0 (socket says 172.21.15.176:40838) 2024-02-19T22:08:32.059 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.009+0000 7f7586ffd700 1 -- 172.21.15.176:0/2060522209 learned_addr learned my addr 172.21.15.176:0/2060522209 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:32.059 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.010+0000 7f7586ffd700 1 -- 172.21.15.176:0/2060522209 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757800b200 con 0x7f75880fafa0 2024-02-19T22:08:32.059 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.010+0000 7f7586ffd700 1 --2- 172.21.15.176:0/2060522209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75880fafa0 0x7f7588068860 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f757800b1d0 tx=0x7f7578006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:32.059 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.010+0000 7f756f7fe700 1 -- 172.21.15.176:0/2060522209 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f757800d8e0 con 0x7f75880fafa0 2024-02-19T22:08:32.059 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.010+0000 7f756f7fe700 1 -- 172.21.15.176:0/2060522209 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f757800da40 con 0x7f75880fafa0 2024-02-19T22:08:32.059 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.010+0000 7f756f7fe700 1 -- 172.21.15.176:0/2060522209 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f757801a710 con 0x7f75880fafa0 2024-02-19T22:08:32.059 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.010+0000 7f758d3c7700 1 -- 172.21.15.176:0/2060522209 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7588068f50 con 0x7f75880fafa0 2024-02-19T22:08:32.059 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.011+0000 7f758d3c7700 1 -- 172.21.15.176:0/2060522209 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75880693e0 con 0x7f75880fafa0 2024-02-19T22:08:32.059 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.012+0000 7f758d3c7700 1 -- 172.21.15.176:0/2060522209 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7588045ad0 con 0x7f75880fafa0 2024-02-19T22:08:32.059 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.012+0000 7f756f7fe700 1 -- 172.21.15.176:0/2060522209 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f757800f070 con 0x7f75880fafa0 2024-02-19T22:08:32.060 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.012+0000 7f756f7fe700 1 --2- 172.21.15.176:0/2060522209 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f7574030060 0x7f7574032520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:32.060 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.013+0000 7f756f7fe700 1 -- 172.21.15.176:0/2060522209 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f75780412e0 con 0x7f75880fafa0 2024-02-19T22:08:32.060 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.013+0000 7f75867fc700 1 -- 172.21.15.176:0/2060522209 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f7574030060 msgr2=0x7f7574032520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/918424985 2024-02-19T22:08:32.060 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.013+0000 7f75867fc700 1 --2- 172.21.15.176:0/2060522209 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f7574030060 0x7f7574032520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:08:32.060 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.020+0000 7f756f7fe700 1 -- 172.21.15.176:0/2060522209 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f757802b8e0 con 0x7f75880fafa0 2024-02-19T22:08:32.214 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.214+0000 7f75867fc700 1 -- 172.21.15.176:0/2060522209 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f7574030060 msgr2=0x7f7574032520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/918424985 2024-02-19T22:08:32.214 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.214+0000 7f75867fc700 1 --2- 172.21.15.176:0/2060522209 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f7574030060 0x7f7574032520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:08:32.225 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.224+0000 7f758d3c7700 1 -- 172.21.15.176:0/2060522209 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f758811b460 con 0x7f75880fafa0 2024-02-19T22:08:32.225 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.224+0000 7f756f7fe700 1 -- 172.21.15.176:0/2060522209 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f7578016020 con 0x7f75880fafa0 2024-02-19T22:08:32.225 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: { 2024-02-19T22:08:32.225 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "epoch": 13, 2024-02-19T22:08:32.225 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "available": true, 2024-02-19T22:08:32.225 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "active_name": "smithi176.faoogw", 2024-02-19T22:08:32.225 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-19T22:08:32.225 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: } 2024-02-19T22:08:32.227 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.226+0000 7f758d3c7700 1 -- 172.21.15.176:0/2060522209 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f7574030060 msgr2=0x7f7574032520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:08:32.227 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.226+0000 7f758d3c7700 1 --2- 172.21.15.176:0/2060522209 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f7574030060 0x7f7574032520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:08:32.227 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.226+0000 7f758d3c7700 1 -- 172.21.15.176:0/2060522209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75880fafa0 msgr2=0x7f7588068860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:32.227 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.226+0000 7f758d3c7700 1 --2- 172.21.15.176:0/2060522209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75880fafa0 0x7f7588068860 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f757800b1d0 tx=0x7f7578006ee0).stop 2024-02-19T22:08:32.227 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.226+0000 7f758d3c7700 1 -- 172.21.15.176:0/2060522209 shutdown_connections 2024-02-19T22:08:32.227 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.226+0000 7f758d3c7700 1 --2- 172.21.15.176:0/2060522209 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7f7574030060 0x7f7574032520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:08:32.227 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.226+0000 7f758d3c7700 1 --2- 172.21.15.176:0/2060522209 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75880fafa0 0x7f7588068860 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:32.228 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.226+0000 7f758d3c7700 1 -- 172.21.15.176:0/2060522209 >> 172.21.15.176:0/2060522209 conn(0x7f75880f5ae0 msgr2=0x7f75880f7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:32.228 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.226+0000 7f758d3c7700 1 -- 172.21.15.176:0/2060522209 shutdown_connections 2024-02-19T22:08:32.228 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:32.226+0000 7f758d3c7700 1 -- 172.21.15.176:0/2060522209 wait complete. 2024-02-19T22:08:32.884 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:32 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:32.224972+0000 mon.smithi176 (mon.0) 89 : audit [DBG] from='client.? 172.21.15.176:0/2060522209' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T22:08:32.969 INFO:teuthology.orchestra.run.smithi176.stderr:Waiting for the mgr to restart... 2024-02-19T22:08:32.969 INFO:teuthology.orchestra.run.smithi176.stderr:Waiting for mgr epoch 13... 2024-02-19T22:08:32.969 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-19T22:08:33.522 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.520+0000 7fb5e43ac700 1 Processor -- start 2024-02-19T22:08:33.522 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.520+0000 7fb5e43ac700 1 -- start start 2024-02-19T22:08:33.522 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.521+0000 7fb5e43ac700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5dc0703d0 0x7fb5dc100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:33.522 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.521+0000 7fb5e43ac700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc0fc8d0 con 0x7fb5dc0703d0 2024-02-19T22:08:33.522 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.521+0000 7fb5e2148700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5dc0703d0 0x7fb5dc100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:33.522 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.521+0000 7fb5e2148700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5dc0703d0 0x7fb5dc100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40850/0 (socket says 172.21.15.176:40850) 2024-02-19T22:08:33.523 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.521+0000 7fb5e2148700 1 -- 172.21.15.176:0/4156639239 learned_addr learned my addr 172.21.15.176:0/4156639239 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:33.523 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.521+0000 7fb5e2148700 1 -- 172.21.15.176:0/4156639239 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc1013d0 con 0x7fb5dc0703d0 2024-02-19T22:08:33.523 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.522+0000 7fb5e2148700 1 --2- 172.21.15.176:0/4156639239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5dc0703d0 0x7fb5dc100ee0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fb5d800b800 tx=0x7fb5d8006ee0).ready entity=mon.0 client_cookie=8b83905826e71c16 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:33.523 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.522+0000 7fb5e1146700 1 -- 172.21.15.176:0/4156639239 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5d8009740 con 0x7fb5dc0703d0 2024-02-19T22:08:33.523 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.522+0000 7fb5e1146700 1 -- 172.21.15.176:0/4156639239 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5d8009d70 con 0x7fb5dc0703d0 2024-02-19T22:08:33.523 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.522+0000 7fb5e1146700 1 -- 172.21.15.176:0/4156639239 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5d8010360 con 0x7fb5dc0703d0 2024-02-19T22:08:33.526 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.522+0000 7fb5e43ac700 1 -- 172.21.15.176:0/4156639239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5dc0703d0 msgr2=0x7fb5dc100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:33.526 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.522+0000 7fb5e43ac700 1 --2- 172.21.15.176:0/4156639239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5dc0703d0 0x7fb5dc100ee0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fb5d800b800 tx=0x7fb5d8006ee0).stop 2024-02-19T22:08:33.526 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.522+0000 7fb5e43ac700 1 -- 172.21.15.176:0/4156639239 shutdown_connections 2024-02-19T22:08:33.526 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.522+0000 7fb5e43ac700 1 --2- 172.21.15.176:0/4156639239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5dc0703d0 0x7fb5dc100ee0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:33.526 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.522+0000 7fb5e43ac700 1 -- 172.21.15.176:0/4156639239 >> 172.21.15.176:0/4156639239 conn(0x7fb5dc0f6520 msgr2=0x7fb5dc0f8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:33.526 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.523+0000 7fb5e43ac700 1 -- 172.21.15.176:0/4156639239 shutdown_connections 2024-02-19T22:08:33.526 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.523+0000 7fb5e43ac700 1 -- 172.21.15.176:0/4156639239 wait complete. 2024-02-19T22:08:33.526 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.523+0000 7fb5e43ac700 1 Processor -- start 2024-02-19T22:08:33.526 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.523+0000 7fb5e43ac700 1 -- start start 2024-02-19T22:08:33.526 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.523+0000 7fb5e43ac700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5dc0703d0 0x7fb5dc10ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:33.526 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.523+0000 7fb5e43ac700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc10d3a0 con 0x7fb5dc0703d0 2024-02-19T22:08:33.527 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.523+0000 7fb5e2148700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5dc0703d0 0x7fb5dc10ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:33.527 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.523+0000 7fb5e2148700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5dc0703d0 0x7fb5dc10ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40858/0 (socket says 172.21.15.176:40858) 2024-02-19T22:08:33.527 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.523+0000 7fb5e2148700 1 -- 172.21.15.176:0/308970216 learned_addr learned my addr 172.21.15.176:0/308970216 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:33.527 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.524+0000 7fb5e2148700 1 -- 172.21.15.176:0/308970216 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d800b200 con 0x7fb5dc0703d0 2024-02-19T22:08:33.527 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.524+0000 7fb5e2148700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5dc0703d0 0x7fb5dc10ceb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fb5d8009a60 tx=0x7fb5d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:33.527 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.524+0000 7fb5caffd700 1 -- 172.21.15.176:0/308970216 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5d8011d00 con 0x7fb5dc0703d0 2024-02-19T22:08:33.527 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.524+0000 7fb5caffd700 1 -- 172.21.15.176:0/308970216 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5d801aa50 con 0x7fb5dc0703d0 2024-02-19T22:08:33.527 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.524+0000 7fb5e43ac700 1 -- 172.21.15.176:0/308970216 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5dc10d5a0 con 0x7fb5dc0703d0 2024-02-19T22:08:33.527 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.524+0000 7fb5caffd700 1 -- 172.21.15.176:0/308970216 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5d801ba30 con 0x7fb5dc0703d0 2024-02-19T22:08:33.527 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.524+0000 7fb5e43ac700 1 -- 172.21.15.176:0/308970216 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5dc10da30 con 0x7fb5dc0703d0 2024-02-19T22:08:33.527 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.525+0000 7fb5caffd700 1 -- 172.21.15.176:0/308970216 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fb5d800f070 con 0x7fb5dc0703d0 2024-02-19T22:08:33.527 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.525+0000 7fb5caffd700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fb5c402ff70 0x7fb5c4032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:33.527 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.525+0000 7fb5caffd700 1 -- 172.21.15.176:0/308970216 --> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5c4032af0 con 0x7fb5c402ff70 2024-02-19T22:08:33.528 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.525+0000 7fb5caffd700 1 -- 172.21.15.176:0/308970216 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb5d8041850 con 0x7fb5dc0703d0 2024-02-19T22:08:33.528 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.525+0000 7fb5e1947700 1 -- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fb5c402ff70 msgr2=0x7fb5c4032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/918424985 2024-02-19T22:08:33.528 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.525+0000 7fb5e1947700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fb5c402ff70 0x7fb5c4032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:08:33.726 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.726+0000 7fb5e1947700 1 -- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fb5c402ff70 msgr2=0x7fb5c4032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/918424985 2024-02-19T22:08:33.726 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:33.726+0000 7fb5e1947700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fb5c402ff70 0x7fb5c4032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:08:34.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.127+0000 7fb5e1947700 1 -- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fb5c402ff70 msgr2=0x7fb5c4032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/918424985 2024-02-19T22:08:34.127 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.127+0000 7fb5e1947700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fb5c402ff70 0x7fb5c4032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T22:08:34.928 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.927+0000 7fb5e1947700 1 -- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fb5c402ff70 msgr2=0x7fb5c4032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.176:6800/918424985 2024-02-19T22:08:34.928 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:34.927+0000 7fb5e1947700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fb5c402ff70 0x7fb5c4032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T22:08:35.826 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.824+0000 7fb5caffd700 1 -- 172.21.15.176:0/308970216 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fb5d801abc0 con 0x7fb5dc0703d0 2024-02-19T22:08:35.826 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.824+0000 7fb5caffd700 1 -- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fb5c402ff70 msgr2=0x7fb5c4032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:08:35.826 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:35.824+0000 7fb5caffd700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fb5c402ff70 0x7fb5c4032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:08:36.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:35.774234+0000 mon.smithi176 (mon.0) 90 : cluster [INF] Active manager daemon smithi176.faoogw restarted 2024-02-19T22:08:36.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:35.774649+0000 mon.smithi176 (mon.0) 91 : cluster [INF] Activating manager daemon smithi176.faoogw 2024-02-19T22:08:36.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:35.776298+0000 mon.smithi176 (mon.0) 92 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-19T22:08:36.833 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.831+0000 7fb5caffd700 1 -- 172.21.15.176:0/308970216 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fb5d8019300 con 0x7fb5dc0703d0 2024-02-19T22:08:36.833 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.831+0000 7fb5caffd700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb5c4033c00 0x7fb5c4035ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:36.833 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.831+0000 7fb5caffd700 1 -- 172.21.15.176:0/308970216 --> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5c4032af0 con 0x7fb5c4033c00 2024-02-19T22:08:36.833 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.831+0000 7fb5e1947700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb5c4033c00 0x7fb5c4035ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:36.833 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.831+0000 7fb5e1947700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb5c4033c00 0x7fb5c4035ff0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb5cc0036d0 tx=0x7fb5cc006ca0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:36.834 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.832+0000 7fb5caffd700 1 -- 172.21.15.176:0/308970216 <== mgr.14162 v2:172.21.15.176:6800/3746885080 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fb5c4032af0 con 0x7fb5c4033c00 2024-02-19T22:08:36.839 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.838+0000 7fb5e43ac700 1 -- 172.21.15.176:0/308970216 --> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb5dc059100 con 0x7fb5c4033c00 2024-02-19T22:08:36.839 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.838+0000 7fb5caffd700 1 -- 172.21.15.176:0/308970216 <== mgr.14162 v2:172.21.15.176:6800/3746885080 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fb5dc059100 con 0x7fb5c4033c00 2024-02-19T22:08:36.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: { 2024-02-19T22:08:36.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-02-19T22:08:36.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: "initialized": true 2024-02-19T22:08:36.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: } 2024-02-19T22:08:36.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.839+0000 7fb5e43ac700 1 -- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb5c4033c00 msgr2=0x7fb5c4035ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:36.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.839+0000 7fb5e43ac700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb5c4033c00 0x7fb5c4035ff0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb5cc0036d0 tx=0x7fb5cc006ca0).stop 2024-02-19T22:08:36.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.839+0000 7fb5e43ac700 1 -- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5dc0703d0 msgr2=0x7fb5dc10ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:36.841 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.839+0000 7fb5e43ac700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5dc0703d0 0x7fb5dc10ceb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fb5d8009a60 tx=0x7fb5d8006ee0).stop 2024-02-19T22:08:36.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.839+0000 7fb5e43ac700 1 -- 172.21.15.176:0/308970216 shutdown_connections 2024-02-19T22:08:36.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.839+0000 7fb5e43ac700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb5c4033c00 0x7fb5c4035ff0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:36.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.839+0000 7fb5e43ac700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb5dc0703d0 0x7fb5dc10ceb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:36.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.839+0000 7fb5e43ac700 1 --2- 172.21.15.176:0/308970216 >> [v2:172.21.15.176:6800/918424985,v1:172.21.15.176:6801/918424985] conn(0x7fb5c402ff70 0x7fb5c4032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:08:36.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.839+0000 7fb5e43ac700 1 -- 172.21.15.176:0/308970216 >> 172.21.15.176:0/308970216 conn(0x7fb5dc0f6520 msgr2=0x7fb5dc0f7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:36.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.839+0000 7fb5e43ac700 1 -- 172.21.15.176:0/308970216 shutdown_connections 2024-02-19T22:08:36.842 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:36.839+0000 7fb5e43ac700 1 -- 172.21.15.176:0/308970216 wait complete. 2024-02-19T22:08:37.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:35.827102+0000 mon.smithi176 (mon.0) 93 : cluster [DBG] mgrmap e14: smithi176.faoogw(active, starting, since 0.0525422s) 2024-02-19T22:08:37.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:35.830207+0000 mon.smithi176 (mon.0) 94 : audit [DBG] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mon metadata", "id": "smithi176"}]: dispatch 2024-02-19T22:08:37.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:35.830379+0000 mon.smithi176 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mgr metadata", "who": "smithi176.faoogw", "id": "smithi176.faoogw"}]: dispatch 2024-02-19T22:08:37.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:35.830546+0000 mon.smithi176 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:08:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:35.830676+0000 mon.smithi176 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:08:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:35.830820+0000 mon.smithi176 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:08:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:35.844195+0000 mon.smithi176 (mon.0) 99 : cluster [INF] Manager daemon smithi176.faoogw is now available 2024-02-19T22:08:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:35.940554+0000 mon.smithi176 (mon.0) 100 : audit [DBG] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:08:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:35.942063+0000 mon.smithi176 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:08:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:35.964202+0000 mon.smithi176 (mon.0) 102 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:08:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:35.968205+0000 mon.smithi176 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:08:37.595 INFO:teuthology.orchestra.run.smithi176.stderr:mgr epoch 13 is available 2024-02-19T22:08:37.595 INFO:teuthology.orchestra.run.smithi176.stderr:Generating a dashboard self-signed certificate... 2024-02-19T22:08:37.595 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-02-19T22:08:37.851 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:36.833816+0000 mgr.smithi176.faoogw (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T22:08:37.851 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:36.834385+0000 mon.smithi176 (mon.0) 104 : cluster [DBG] mgrmap e15: smithi176.faoogw(active, since 1.05982s) 2024-02-19T22:08:37.851 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:36.840147+0000 mgr.smithi176.faoogw (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T22:08:37.852 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:37.252988+0000 mon.smithi176 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:38.335 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.234+0000 7f206bdf1700 1 Processor -- start 2024-02-19T22:08:38.336 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.234+0000 7f206bdf1700 1 -- start start 2024-02-19T22:08:38.336 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.234+0000 7f206bdf1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20640fad90 0x7f20640fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:38.336 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.234+0000 7f206bdf1700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20640fb6a0 con 0x7f20640fad90 2024-02-19T22:08:38.336 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.234+0000 7f2069b8d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20640fad90 0x7f20640fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:38.336 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.234+0000 7f2069b8d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20640fad90 0x7f20640fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:42848/0 (socket says 172.21.15.176:42848) 2024-02-19T22:08:38.336 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.234+0000 7f2069b8d700 1 -- 172.21.15.176:0/2048171513 learned_addr learned my addr 172.21.15.176:0/2048171513 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:38.336 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.235+0000 7f2069b8d700 1 -- 172.21.15.176:0/2048171513 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20640fb7e0 con 0x7f20640fad90 2024-02-19T22:08:38.336 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.235+0000 7f2069b8d700 1 --2- 172.21.15.176:0/2048171513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20640fad90 0x7f20640fb1b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f206000b5f0 tx=0x7f2060006ee0).ready entity=mon.0 client_cookie=a204ad1f39f62be server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:38.336 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.235+0000 7f2068b8b700 1 -- 172.21.15.176:0/2048171513 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f206000d600 con 0x7f20640fad90 2024-02-19T22:08:38.336 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.235+0000 7f2068b8b700 1 -- 172.21.15.176:0/2048171513 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f206000dc30 con 0x7f20640fad90 2024-02-19T22:08:38.336 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.235+0000 7f2068b8b700 1 -- 172.21.15.176:0/2048171513 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2060011d30 con 0x7f20640fad90 2024-02-19T22:08:38.336 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.235+0000 7f206bdf1700 1 -- 172.21.15.176:0/2048171513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20640fad90 msgr2=0x7f20640fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:38.337 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.235+0000 7f206bdf1700 1 --2- 172.21.15.176:0/2048171513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20640fad90 0x7f20640fb1b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f206000b5f0 tx=0x7f2060006ee0).stop 2024-02-19T22:08:38.337 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.235+0000 7f206bdf1700 1 -- 172.21.15.176:0/2048171513 shutdown_connections 2024-02-19T22:08:38.337 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.235+0000 7f206bdf1700 1 --2- 172.21.15.176:0/2048171513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20640fad90 0x7f20640fb1b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:38.337 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.235+0000 7f206bdf1700 1 -- 172.21.15.176:0/2048171513 >> 172.21.15.176:0/2048171513 conn(0x7f20640f6530 msgr2=0x7f20640f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:38.337 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.235+0000 7f206bdf1700 1 -- 172.21.15.176:0/2048171513 shutdown_connections 2024-02-19T22:08:38.337 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.236+0000 7f206bdf1700 1 -- 172.21.15.176:0/2048171513 wait complete. 2024-02-19T22:08:38.337 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.236+0000 7f206bdf1700 1 Processor -- start 2024-02-19T22:08:38.337 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.236+0000 7f206bdf1700 1 -- start start 2024-02-19T22:08:38.337 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.236+0000 7f206bdf1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20640fad90 0x7f206410abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:38.337 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.236+0000 7f206bdf1700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206410b0a0 con 0x7f20640fad90 2024-02-19T22:08:38.337 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.236+0000 7f2069b8d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20640fad90 0x7f206410abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:38.337 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.236+0000 7f2069b8d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20640fad90 0x7f206410abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:42852/0 (socket says 172.21.15.176:42852) 2024-02-19T22:08:38.337 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.236+0000 7f2069b8d700 1 -- 172.21.15.176:0/3558324267 learned_addr learned my addr 172.21.15.176:0/3558324267 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:38.338 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.237+0000 7f2069b8d700 1 -- 172.21.15.176:0/3558324267 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206000b200 con 0x7f20640fad90 2024-02-19T22:08:38.338 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.237+0000 7f2069b8d700 1 --2- 172.21.15.176:0/3558324267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20640fad90 0x7f206410abb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f206000e040 tx=0x7f2060006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:38.338 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.237+0000 7f20527fc700 1 -- 172.21.15.176:0/3558324267 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f206000d9e0 con 0x7f20640fad90 2024-02-19T22:08:38.338 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.237+0000 7f206bdf1700 1 -- 172.21.15.176:0/3558324267 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206410b2a0 con 0x7f20640fad90 2024-02-19T22:08:38.338 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.237+0000 7f20527fc700 1 -- 172.21.15.176:0/3558324267 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f206001b8e0 con 0x7f20640fad90 2024-02-19T22:08:38.338 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.237+0000 7f20527fc700 1 -- 172.21.15.176:0/3558324267 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f206001a880 con 0x7f20640fad90 2024-02-19T22:08:38.338 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.237+0000 7f206bdf1700 1 -- 172.21.15.176:0/3558324267 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f206410b730 con 0x7f20640fad90 2024-02-19T22:08:38.338 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.238+0000 7f20527fc700 1 -- 172.21.15.176:0/3558324267 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f206000f070 con 0x7f20640fad90 2024-02-19T22:08:38.338 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.238+0000 7f20527fc700 1 --2- 172.21.15.176:0/3558324267 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f204c02fe30 0x7f204c0322f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:38.338 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.238+0000 7f206bdf1700 1 -- 172.21.15.176:0/3558324267 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2064045ad0 con 0x7f20640fad90 2024-02-19T22:08:38.338 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.238+0000 7f20527fc700 1 -- 172.21.15.176:0/3558324267 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f20600414a0 con 0x7f20640fad90 2024-02-19T22:08:38.338 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.238+0000 7f206938c700 1 --2- 172.21.15.176:0/3558324267 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f204c02fe30 0x7f204c0322f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:38.338 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.238+0000 7f206938c700 1 --2- 172.21.15.176:0/3558324267 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f204c02fe30 0x7f204c0322f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f20540096a0 tx=0x7f2054006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:38.338 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.242+0000 7f20527fc700 1 -- 172.21.15.176:0/3558324267 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2060016070 con 0x7f20640fad90 2024-02-19T22:08:38.339 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.252+0000 7f20527fc700 1 -- 172.21.15.176:0/3558324267 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f206001b410 con 0x7f20640fad90 2024-02-19T22:08:38.407 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.406+0000 7f206bdf1700 1 -- 172.21.15.176:0/3558324267 --> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f206410cdf0 con 0x7f204c02fe30 2024-02-19T22:08:38.487 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.486+0000 7f20527fc700 1 -- 172.21.15.176:0/3558324267 <== mgr.14162 v2:172.21.15.176:6800/3746885080 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f206410cdf0 con 0x7f204c02fe30 2024-02-19T22:08:38.487 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-19T22:08:38.489 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.487+0000 7f206bdf1700 1 -- 172.21.15.176:0/3558324267 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f204c02fe30 msgr2=0x7f204c0322f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:38.489 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.487+0000 7f206bdf1700 1 --2- 172.21.15.176:0/3558324267 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f204c02fe30 0x7f204c0322f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f20540096a0 tx=0x7f2054006b40).stop 2024-02-19T22:08:38.489 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.487+0000 7f206bdf1700 1 -- 172.21.15.176:0/3558324267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20640fad90 msgr2=0x7f206410abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:38.489 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.487+0000 7f206bdf1700 1 --2- 172.21.15.176:0/3558324267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20640fad90 0x7f206410abb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f206000e040 tx=0x7f2060006ee0).stop 2024-02-19T22:08:38.490 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.487+0000 7f206bdf1700 1 -- 172.21.15.176:0/3558324267 shutdown_connections 2024-02-19T22:08:38.490 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.488+0000 7f206bdf1700 1 --2- 172.21.15.176:0/3558324267 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f204c02fe30 0x7f204c0322f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:38.490 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.488+0000 7f206bdf1700 1 --2- 172.21.15.176:0/3558324267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20640fad90 0x7f206410abb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:38.490 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.488+0000 7f206bdf1700 1 -- 172.21.15.176:0/3558324267 >> 172.21.15.176:0/3558324267 conn(0x7f20640f6530 msgr2=0x7f2064070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:38.490 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.488+0000 7f206bdf1700 1 -- 172.21.15.176:0/3558324267 shutdown_connections 2024-02-19T22:08:38.490 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:38.488+0000 7f206bdf1700 1 -- 172.21.15.176:0/3558324267 wait complete. 2024-02-19T22:08:39.180 INFO:teuthology.orchestra.run.smithi176.stderr:Creating initial admin user... 2024-02-19T22:08:39.180 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpfgdbs0xl:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-02-19T22:08:39.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:38.254589+0000 mon.smithi176 (mon.0) 106 : cluster [DBG] mgrmap e16: smithi176.faoogw(active, since 2s) 2024-02-19T22:08:39.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:38.408768+0000 mgr.smithi176.faoogw (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:08:39.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:38.485644+0000 mon.smithi176 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:39.634 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:38.487358+0000 mon.smithi176 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:39.741 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.736+0000 7f71c1f07700 1 Processor -- start 2024-02-19T22:08:39.741 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.736+0000 7f71c1f07700 1 -- start start 2024-02-19T22:08:39.741 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.737+0000 7f71c1f07700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71bc0fbe80 0x7f71bc0fc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:39.741 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.737+0000 7f71c1f07700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc0fc790 con 0x7f71bc0fbe80 2024-02-19T22:08:39.741 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.737+0000 7f71bb7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71bc0fbe80 0x7f71bc0fc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:39.741 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.737+0000 7f71bb7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71bc0fbe80 0x7f71bc0fc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:42856/0 (socket says 172.21.15.176:42856) 2024-02-19T22:08:39.741 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.737+0000 7f71bb7fe700 1 -- 172.21.15.176:0/3265441546 learned_addr learned my addr 172.21.15.176:0/3265441546 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:39.741 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.737+0000 7f71bb7fe700 1 -- 172.21.15.176:0/3265441546 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc0fc8d0 con 0x7f71bc0fbe80 2024-02-19T22:08:39.741 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.737+0000 7f71bb7fe700 1 --2- 172.21.15.176:0/3265441546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71bc0fbe80 0x7f71bc0fc2a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f71b000b5f0 tx=0x7f71b0006ee0).ready entity=mon.0 client_cookie=e3c06051c6e947cf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:39.741 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.737+0000 7f71ba7fc700 1 -- 172.21.15.176:0/3265441546 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b000d600 con 0x7f71bc0fbe80 2024-02-19T22:08:39.741 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.737+0000 7f71ba7fc700 1 -- 172.21.15.176:0/3265441546 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71b000dc30 con 0x7f71bc0fbe80 2024-02-19T22:08:39.741 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.737+0000 7f71ba7fc700 1 -- 172.21.15.176:0/3265441546 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b0011d30 con 0x7f71bc0fbe80 2024-02-19T22:08:39.742 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.738+0000 7f71c1f07700 1 -- 172.21.15.176:0/3265441546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71bc0fbe80 msgr2=0x7f71bc0fc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:39.742 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.738+0000 7f71c1f07700 1 --2- 172.21.15.176:0/3265441546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71bc0fbe80 0x7f71bc0fc2a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f71b000b5f0 tx=0x7f71b0006ee0).stop 2024-02-19T22:08:39.742 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.738+0000 7f71c1f07700 1 -- 172.21.15.176:0/3265441546 shutdown_connections 2024-02-19T22:08:39.742 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.738+0000 7f71c1f07700 1 --2- 172.21.15.176:0/3265441546 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71bc0fbe80 0x7f71bc0fc2a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:39.742 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.738+0000 7f71c1f07700 1 -- 172.21.15.176:0/3265441546 >> 172.21.15.176:0/3265441546 conn(0x7f71bc0f7640 msgr2=0x7f71bc0f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:39.742 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.738+0000 7f71c1f07700 1 -- 172.21.15.176:0/3265441546 shutdown_connections 2024-02-19T22:08:39.742 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.738+0000 7f71c1f07700 1 -- 172.21.15.176:0/3265441546 wait complete. 2024-02-19T22:08:39.742 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.738+0000 7f71c1f07700 1 Processor -- start 2024-02-19T22:08:39.742 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.739+0000 7f71c1f07700 1 -- start start 2024-02-19T22:08:39.742 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.739+0000 7f71c1f07700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71bc0fbe80 0x7f71bc072570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:39.742 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.739+0000 7f71c1f07700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc0709f0 con 0x7f71bc0fbe80 2024-02-19T22:08:39.742 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.739+0000 7f71bb7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71bc0fbe80 0x7f71bc072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:39.742 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.739+0000 7f71bb7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71bc0fbe80 0x7f71bc072570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:42860/0 (socket says 172.21.15.176:42860) 2024-02-19T22:08:39.742 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.739+0000 7f71bb7fe700 1 -- 172.21.15.176:0/2496919764 learned_addr learned my addr 172.21.15.176:0/2496919764 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:39.743 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.739+0000 7f71bb7fe700 1 -- 172.21.15.176:0/2496919764 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b000b200 con 0x7f71bc0fbe80 2024-02-19T22:08:39.743 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.739+0000 7f71bb7fe700 1 --2- 172.21.15.176:0/2496919764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71bc0fbe80 0x7f71bc072570 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f71b0006b80 tx=0x7f71b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:39.743 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.739+0000 7f71a3fff700 1 -- 172.21.15.176:0/2496919764 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b000da60 con 0x7f71bc0fbe80 2024-02-19T22:08:39.743 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.739+0000 7f71a3fff700 1 -- 172.21.15.176:0/2496919764 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71b001b8e0 con 0x7f71bc0fbe80 2024-02-19T22:08:39.743 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.739+0000 7f71a3fff700 1 -- 172.21.15.176:0/2496919764 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b001a960 con 0x7f71bc0fbe80 2024-02-19T22:08:39.743 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.739+0000 7f71c1f07700 1 -- 172.21.15.176:0/2496919764 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71bc070b90 con 0x7f71bc0fbe80 2024-02-19T22:08:39.743 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.740+0000 7f71c1f07700 1 -- 172.21.15.176:0/2496919764 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71bc071020 con 0x7f71bc0fbe80 2024-02-19T22:08:39.744 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.741+0000 7f71c1f07700 1 -- 172.21.15.176:0/2496919764 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71bc045ad0 con 0x7f71bc0fbe80 2024-02-19T22:08:39.744 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.741+0000 7f71a3fff700 1 -- 172.21.15.176:0/2496919764 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f71b000f070 con 0x7f71bc0fbe80 2024-02-19T22:08:39.744 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.741+0000 7f71a3fff700 1 --2- 172.21.15.176:0/2496919764 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f71a802ffc0 0x7f71a8032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:39.744 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.741+0000 7f71a3fff700 1 -- 172.21.15.176:0/2496919764 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f71b0041c00 con 0x7f71bc0fbe80 2024-02-19T22:08:39.744 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.741+0000 7f71baffd700 1 --2- 172.21.15.176:0/2496919764 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f71a802ffc0 0x7f71a8032480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:39.744 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.742+0000 7f71baffd700 1 --2- 172.21.15.176:0/2496919764 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f71a802ffc0 0x7f71a8032480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f71ac0096a0 tx=0x7f71ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:39.746 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.745+0000 7f71a3fff700 1 -- 172.21.15.176:0/2496919764 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71b0023b50 con 0x7f71bc0fbe80 2024-02-19T22:08:39.912 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:39.911+0000 7f71c1f07700 1 -- 172.21.15.176:0/2496919764 --> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f71bc058f20 con 0x7f71a802ffc0 2024-02-19T22:08:40.132 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:40.131+0000 7f71a3fff700 1 -- 172.21.15.176:0/2496919764 <== mgr.14162 v2:172.21.15.176:6800/3746885080 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f71bc058f20 con 0x7f71a802ffc0 2024-02-19T22:08:40.133 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$VDfxxMYFjbpJfxnalBoJwO5j2mnzQXIZ5Ps2jThYdb9LMfVQapbee", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708380520, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-19T22:08:40.135 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:40.133+0000 7f71c1f07700 1 -- 172.21.15.176:0/2496919764 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f71a802ffc0 msgr2=0x7f71a8032480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:40.135 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:40.133+0000 7f71c1f07700 1 --2- 172.21.15.176:0/2496919764 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f71a802ffc0 0x7f71a8032480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f71ac0096a0 tx=0x7f71ac006b40).stop 2024-02-19T22:08:40.135 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:40.133+0000 7f71c1f07700 1 -- 172.21.15.176:0/2496919764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71bc0fbe80 msgr2=0x7f71bc072570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:40.135 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:40.133+0000 7f71c1f07700 1 --2- 172.21.15.176:0/2496919764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71bc0fbe80 0x7f71bc072570 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f71b0006b80 tx=0x7f71b0006ee0).stop 2024-02-19T22:08:40.135 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:40.133+0000 7f71c1f07700 1 -- 172.21.15.176:0/2496919764 shutdown_connections 2024-02-19T22:08:40.135 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:40.133+0000 7f71c1f07700 1 --2- 172.21.15.176:0/2496919764 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f71a802ffc0 0x7f71a8032480 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:40.135 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:40.133+0000 7f71c1f07700 1 --2- 172.21.15.176:0/2496919764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71bc0fbe80 0x7f71bc072570 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:40.135 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:40.133+0000 7f71c1f07700 1 -- 172.21.15.176:0/2496919764 >> 172.21.15.176:0/2496919764 conn(0x7f71bc0f7640 msgr2=0x7f71bc0f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:40.135 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:40.133+0000 7f71c1f07700 1 -- 172.21.15.176:0/2496919764 shutdown_connections 2024-02-19T22:08:40.135 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:40.133+0000 7f71c1f07700 1 -- 172.21.15.176:0/2496919764 wait complete. 2024-02-19T22:08:40.798 INFO:teuthology.orchestra.run.smithi176.stderr:Fetching dashboard port number... 2024-02-19T22:08:40.798 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0dj93obq:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpnyo8bwb8:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-02-19T22:08:41.369 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.340+0000 7f7132b75700 1 Processor -- start 2024-02-19T22:08:41.369 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.340+0000 7f7132b75700 1 -- start start 2024-02-19T22:08:41.369 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.340+0000 7f7132b75700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f712c0fad90 0x7f712c0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:41.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.340+0000 7f7132b75700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c0fb6a0 con 0x7f712c0fad90 2024-02-19T22:08:41.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.340+0000 7f7130911700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f712c0fad90 0x7f712c0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:41.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.340+0000 7f7130911700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f712c0fad90 0x7f712c0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:42864/0 (socket says 172.21.15.176:42864) 2024-02-19T22:08:41.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.340+0000 7f7130911700 1 -- 172.21.15.176:0/3399085726 learned_addr learned my addr 172.21.15.176:0/3399085726 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:41.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.341+0000 7f7130911700 1 -- 172.21.15.176:0/3399085726 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712c0fb7e0 con 0x7f712c0fad90 2024-02-19T22:08:41.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.341+0000 7f7130911700 1 --2- 172.21.15.176:0/3399085726 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f712c0fad90 0x7f712c0fb1b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f712000b5f0 tx=0x7f7120006ee0).ready entity=mon.0 client_cookie=b1576e9e00e396c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:41.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.341+0000 7f712b7fe700 1 -- 172.21.15.176:0/3399085726 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f712000d600 con 0x7f712c0fad90 2024-02-19T22:08:41.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.341+0000 7f712b7fe700 1 -- 172.21.15.176:0/3399085726 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f712000dc30 con 0x7f712c0fad90 2024-02-19T22:08:41.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.341+0000 7f712b7fe700 1 -- 172.21.15.176:0/3399085726 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7120011d30 con 0x7f712c0fad90 2024-02-19T22:08:41.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.341+0000 7f7132b75700 1 -- 172.21.15.176:0/3399085726 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f712c0fad90 msgr2=0x7f712c0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:41.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.341+0000 7f7132b75700 1 --2- 172.21.15.176:0/3399085726 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f712c0fad90 0x7f712c0fb1b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f712000b5f0 tx=0x7f7120006ee0).stop 2024-02-19T22:08:41.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.341+0000 7f7132b75700 1 -- 172.21.15.176:0/3399085726 shutdown_connections 2024-02-19T22:08:41.370 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.341+0000 7f7132b75700 1 --2- 172.21.15.176:0/3399085726 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f712c0fad90 0x7f712c0fb1b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:41.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.341+0000 7f7132b75700 1 -- 172.21.15.176:0/3399085726 >> 172.21.15.176:0/3399085726 conn(0x7f712c0f6530 msgr2=0x7f712c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:41.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.341+0000 7f7132b75700 1 -- 172.21.15.176:0/3399085726 shutdown_connections 2024-02-19T22:08:41.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.341+0000 7f7132b75700 1 -- 172.21.15.176:0/3399085726 wait complete. 2024-02-19T22:08:41.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.342+0000 7f7132b75700 1 Processor -- start 2024-02-19T22:08:41.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.342+0000 7f7132b75700 1 -- start start 2024-02-19T22:08:41.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.342+0000 7f7132b75700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f712c0fad90 0x7f712c108940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:41.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.342+0000 7f7132b75700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c108e30 con 0x7f712c0fad90 2024-02-19T22:08:41.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.342+0000 7f7130911700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f712c0fad90 0x7f712c108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:41.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.342+0000 7f7130911700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f712c0fad90 0x7f712c108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:42876/0 (socket says 172.21.15.176:42876) 2024-02-19T22:08:41.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.342+0000 7f7130911700 1 -- 172.21.15.176:0/4023377572 learned_addr learned my addr 172.21.15.176:0/4023377572 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:41.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.343+0000 7f7130911700 1 -- 172.21.15.176:0/4023377572 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712000b200 con 0x7f712c0fad90 2024-02-19T22:08:41.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.343+0000 7f7130911700 1 --2- 172.21.15.176:0/4023377572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f712c0fad90 0x7f712c108940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f7120006b80 tx=0x7f7120006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:41.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.343+0000 7f71297fa700 1 -- 172.21.15.176:0/4023377572 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f712000da60 con 0x7f712c0fad90 2024-02-19T22:08:41.371 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.343+0000 7f71297fa700 1 -- 172.21.15.176:0/4023377572 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f712001b8e0 con 0x7f712c0fad90 2024-02-19T22:08:41.372 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.343+0000 7f7132b75700 1 -- 172.21.15.176:0/4023377572 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712c109030 con 0x7f712c0fad90 2024-02-19T22:08:41.372 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.343+0000 7f71297fa700 1 -- 172.21.15.176:0/4023377572 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f712001a960 con 0x7f712c0fad90 2024-02-19T22:08:41.372 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.343+0000 7f7132b75700 1 -- 172.21.15.176:0/4023377572 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c1094c0 con 0x7f712c0fad90 2024-02-19T22:08:41.372 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.344+0000 7f7132b75700 1 -- 172.21.15.176:0/4023377572 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712c045ad0 con 0x7f712c0fad90 2024-02-19T22:08:41.372 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.344+0000 7f71297fa700 1 -- 172.21.15.176:0/4023377572 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f712000f070 con 0x7f712c0fad90 2024-02-19T22:08:41.372 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.344+0000 7f71297fa700 1 --2- 172.21.15.176:0/4023377572 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f71100321c0 0x7f7110034680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:41.372 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.344+0000 7f71297fa700 1 -- 172.21.15.176:0/4023377572 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7120041480 con 0x7f712c0fad90 2024-02-19T22:08:41.372 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.344+0000 7f712bfff700 1 --2- 172.21.15.176:0/4023377572 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f71100321c0 0x7f7110034680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:41.372 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.345+0000 7f712bfff700 1 --2- 172.21.15.176:0/4023377572 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f71100321c0 0x7f7110034680 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f71180096a0 tx=0x7f7118006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:41.372 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.348+0000 7f71297fa700 1 -- 172.21.15.176:0/4023377572 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7120019c90 con 0x7f712c0fad90 2024-02-19T22:08:41.384 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:39.913737+0000 mgr.smithi176.faoogw (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:08:41.384 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:40.132926+0000 mon.smithi176 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:41.501 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.499+0000 7f7132b75700 1 -- 172.21.15.176:0/4023377572 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f712c102890 con 0x7f712c0fad90 2024-02-19T22:08:41.501 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.500+0000 7f71297fa700 1 -- 172.21.15.176:0/4023377572 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f7120019ab0 con 0x7f712c0fad90 2024-02-19T22:08:41.501 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 8443 2024-02-19T22:08:41.503 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.501+0000 7f7132b75700 1 -- 172.21.15.176:0/4023377572 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f71100321c0 msgr2=0x7f7110034680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:41.503 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.501+0000 7f7132b75700 1 --2- 172.21.15.176:0/4023377572 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f71100321c0 0x7f7110034680 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f71180096a0 tx=0x7f7118006b40).stop 2024-02-19T22:08:41.503 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.501+0000 7f7132b75700 1 -- 172.21.15.176:0/4023377572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f712c0fad90 msgr2=0x7f712c108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:41.503 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.501+0000 7f7132b75700 1 --2- 172.21.15.176:0/4023377572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f712c0fad90 0x7f712c108940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f7120006b80 tx=0x7f7120006ee0).stop 2024-02-19T22:08:41.503 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.501+0000 7f7132b75700 1 -- 172.21.15.176:0/4023377572 shutdown_connections 2024-02-19T22:08:41.503 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.501+0000 7f7132b75700 1 --2- 172.21.15.176:0/4023377572 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f71100321c0 0x7f7110034680 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:41.504 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.501+0000 7f7132b75700 1 --2- 172.21.15.176:0/4023377572 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f712c0fad90 0x7f712c108940 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:41.504 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.501+0000 7f7132b75700 1 -- 172.21.15.176:0/4023377572 >> 172.21.15.176:0/4023377572 conn(0x7f712c0f6530 msgr2=0x7f712c0ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:41.504 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.502+0000 7f7132b75700 1 -- 172.21.15.176:0/4023377572 shutdown_connections 2024-02-19T22:08:41.504 INFO:teuthology.orchestra.run.smithi176.stderr:/usr/bin/ceph: 2024-02-19T22:08:41.502+0000 7f7132b75700 1 -- 172.21.15.176:0/4023377572 wait complete. 2024-02-19T22:08:42.238 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:08:42.241 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:42 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:41.501610+0000 mon.smithi176 (mon.0) 110 : audit [DBG] from='client.? 172.21.15.176:0/4023377572' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-19T22:08:42.247 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: disabled 2024-02-19T22:08:42.248 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:08:42.254 INFO:teuthology.orchestra.run.smithi176.stderr:systemctl: inactive 2024-02-19T22:08:42.255 INFO:teuthology.orchestra.run.smithi176.stderr:firewalld.service is not enabled 2024-02-19T22:08:42.255 INFO:teuthology.orchestra.run.smithi176.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-19T22:08:42.258 INFO:teuthology.orchestra.run.smithi176.stderr:Ceph Dashboard is now available at: 2024-02-19T22:08:42.258 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-02-19T22:08:42.258 INFO:teuthology.orchestra.run.smithi176.stderr: URL: https://smithi176.front.sepia.ceph.com:8443/ 2024-02-19T22:08:42.258 INFO:teuthology.orchestra.run.smithi176.stderr: User: admin 2024-02-19T22:08:42.258 INFO:teuthology.orchestra.run.smithi176.stderr: Password: jnp1t0h8jx 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr:You can access the Ceph CLI with: 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr: ceph telemetry on 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr:For more information see: 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr:Bootstrap complete. 2024-02-19T22:08:42.259 INFO:teuthology.orchestra.run.smithi176.stderr:Releasing lock 139877961636328 on /run/cephadm/36a8f1ec-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:08:42.260 INFO:teuthology.orchestra.run.smithi176.stderr:Lock 139877961636328 released on /run/cephadm/36a8f1ec-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:08:42.291 INFO:tasks.cephadm:Fetching config... 2024-02-19T22:08:42.291 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-19T22:08:42.291 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-19T22:08:42.351 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-19T22:08:42.351 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-19T22:08:42.351 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-19T22:08:42.408 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-19T22:08:42.409 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-19T22:08:42.409 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/mon.smithi176/keyring of=/dev/stdout 2024-02-19T22:08:42.476 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-19T22:08:42.476 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-19T22:08:42.476 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-19T22:08:42.534 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-19T22:08:42.535 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwHsNZYEucPRxwPXghnjq8kZ2PKOCEiilJi9d8hHw/TyriGZ679M7DS5IuaOIIuCtBTY+dfMO8G5Yp4qIdpRMq0/1TZbINCvaVNmk7DgRmF0XcwfkbxHlo5Lw6NHG1/GyUv/8yJ1T8zmsyXbn5KXHpMINSnigvhjJbvX6Gzz8Lyr5aq/XOFBsBlFJzYKQHKmyCo1rhvBnlxcpB3NZ3wRGNqJetp0JsbQ36YFOGHjPlnLnvkcGOS7JD53jUVvxuoGMtvNwGhaunh028xgvCXMAV+FC8KuntEmJsATnbAZ7VwNWS2rDm3Wa6eMGGvmquQmMRuIh9+1UTZHWG7ylpNsqbOOTtfKNBOYMmvU6vj7RhCHh9VjkCJNz1AbNINv/ArfJt2/uRQ2HBJ8kZNWl6N1WFCtJ+tbjvQv3hggHxs3gndcchyJXpQ7GO2tok/3ePxwRKyCLtajHtT6OvdICJuF0Xye7P/xEFXMcpuLuiKyYJCMVkABjJI2lGtF553vbPtwc= ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T22:08:42.611 INFO:teuthology.orchestra.run.smithi176.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwHsNZYEucPRxwPXghnjq8kZ2PKOCEiilJi9d8hHw/TyriGZ679M7DS5IuaOIIuCtBTY+dfMO8G5Yp4qIdpRMq0/1TZbINCvaVNmk7DgRmF0XcwfkbxHlo5Lw6NHG1/GyUv/8yJ1T8zmsyXbn5KXHpMINSnigvhjJbvX6Gzz8Lyr5aq/XOFBsBlFJzYKQHKmyCo1rhvBnlxcpB3NZ3wRGNqJetp0JsbQ36YFOGHjPlnLnvkcGOS7JD53jUVvxuoGMtvNwGhaunh028xgvCXMAV+FC8KuntEmJsATnbAZ7VwNWS2rDm3Wa6eMGGvmquQmMRuIh9+1UTZHWG7ylpNsqbOOTtfKNBOYMmvU6vj7RhCHh9VjkCJNz1AbNINv/ArfJt2/uRQ2HBJ8kZNWl6N1WFCtJ+tbjvQv3hggHxs3gndcchyJXpQ7GO2tok/3ePxwRKyCLtajHtT6OvdICJuF0Xye7P/xEFXMcpuLuiKyYJCMVkABjJI2lGtF553vbPtwc= ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:08:42.622 DEBUG:teuthology.orchestra.run.smithi184:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwHsNZYEucPRxwPXghnjq8kZ2PKOCEiilJi9d8hHw/TyriGZ679M7DS5IuaOIIuCtBTY+dfMO8G5Yp4qIdpRMq0/1TZbINCvaVNmk7DgRmF0XcwfkbxHlo5Lw6NHG1/GyUv/8yJ1T8zmsyXbn5KXHpMINSnigvhjJbvX6Gzz8Lyr5aq/XOFBsBlFJzYKQHKmyCo1rhvBnlxcpB3NZ3wRGNqJetp0JsbQ36YFOGHjPlnLnvkcGOS7JD53jUVvxuoGMtvNwGhaunh028xgvCXMAV+FC8KuntEmJsATnbAZ7VwNWS2rDm3Wa6eMGGvmquQmMRuIh9+1UTZHWG7ylpNsqbOOTtfKNBOYMmvU6vj7RhCHh9VjkCJNz1AbNINv/ArfJt2/uRQ2HBJ8kZNWl6N1WFCtJ+tbjvQv3hggHxs3gndcchyJXpQ7GO2tok/3ePxwRKyCLtajHtT6OvdICJuF0Xye7P/xEFXMcpuLuiKyYJCMVkABjJI2lGtF553vbPtwc= ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T22:08:42.667 INFO:teuthology.orchestra.run.smithi184.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwHsNZYEucPRxwPXghnjq8kZ2PKOCEiilJi9d8hHw/TyriGZ679M7DS5IuaOIIuCtBTY+dfMO8G5Yp4qIdpRMq0/1TZbINCvaVNmk7DgRmF0XcwfkbxHlo5Lw6NHG1/GyUv/8yJ1T8zmsyXbn5KXHpMINSnigvhjJbvX6Gzz8Lyr5aq/XOFBsBlFJzYKQHKmyCo1rhvBnlxcpB3NZ3wRGNqJetp0JsbQ36YFOGHjPlnLnvkcGOS7JD53jUVvxuoGMtvNwGhaunh028xgvCXMAV+FC8KuntEmJsATnbAZ7VwNWS2rDm3Wa6eMGGvmquQmMRuIh9+1UTZHWG7ylpNsqbOOTtfKNBOYMmvU6vj7RhCHh9VjkCJNz1AbNINv/ArfJt2/uRQ2HBJ8kZNWl6N1WFCtJ+tbjvQv3hggHxs3gndcchyJXpQ7GO2tok/3ePxwRKyCLtajHtT6OvdICJuF0Xye7P/xEFXMcpuLuiKyYJCMVkABjJI2lGtF553vbPtwc= ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:08:42.679 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-19T22:08:43.492 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:42.134913+0000 mon.smithi176 (mon.0) 111 : cluster [DBG] mgrmap e17: smithi176.faoogw(active, since 6s) 2024-02-19T22:08:45.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.417+0000 7effeb59e700 1 -- 172.21.15.176:0/1333174117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effec0601e0 msgr2=0x7effec0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:45.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.417+0000 7effeb59e700 1 --2- 172.21.15.176:0/1333174117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effec0601e0 0x7effec0f8f20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7effdc009fb0 tx=0x7effdc009b30).stop 2024-02-19T22:08:45.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.418+0000 7effeb59e700 1 -- 172.21.15.176:0/1333174117 shutdown_connections 2024-02-19T22:08:45.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.418+0000 7effeb59e700 1 --2- 172.21.15.176:0/1333174117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effec0601e0 0x7effec0f8f20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:45.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.418+0000 7effeb59e700 1 -- 172.21.15.176:0/1333174117 >> 172.21.15.176:0/1333174117 conn(0x7effec0f09a0 msgr2=0x7effec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:45.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.418+0000 7effeb59e700 1 -- 172.21.15.176:0/1333174117 shutdown_connections 2024-02-19T22:08:45.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.418+0000 7effeb59e700 1 -- 172.21.15.176:0/1333174117 wait complete. 2024-02-19T22:08:45.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.418+0000 7effeb59e700 1 Processor -- start 2024-02-19T22:08:45.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.418+0000 7effeb59e700 1 -- start start 2024-02-19T22:08:45.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.418+0000 7effeb59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effec0601e0 0x7effec1065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:45.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.418+0000 7effeb59e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec106ad0 con 0x7effec0601e0 2024-02-19T22:08:45.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.418+0000 7effea59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effec0601e0 0x7effec1065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:45.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.418+0000 7effea59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effec0601e0 0x7effec1065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:35882/0 (socket says 172.21.15.176:35882) 2024-02-19T22:08:45.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.418+0000 7effea59c700 1 -- 172.21.15.176:0/3665010026 learned_addr learned my addr 172.21.15.176:0/3665010026 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:08:45.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.419+0000 7effea59c700 1 -- 172.21.15.176:0/3665010026 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc005040 con 0x7effec0601e0 2024-02-19T22:08:45.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.419+0000 7effea59c700 1 --2- 172.21.15.176:0/3665010026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effec0601e0 0x7effec1065e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7effdc005010 tx=0x7effdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:45.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.419+0000 7effdaffd700 1 -- 172.21.15.176:0/3665010026 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effdc009320 con 0x7effec0601e0 2024-02-19T22:08:45.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.419+0000 7effdaffd700 1 -- 172.21.15.176:0/3665010026 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effdc013380 con 0x7effec0601e0 2024-02-19T22:08:45.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.419+0000 7effeb59e700 1 -- 172.21.15.176:0/3665010026 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effec106cd0 con 0x7effec0601e0 2024-02-19T22:08:45.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.419+0000 7effdaffd700 1 -- 172.21.15.176:0/3665010026 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effdc014e50 con 0x7effec0601e0 2024-02-19T22:08:45.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.419+0000 7effeb59e700 1 -- 172.21.15.176:0/3665010026 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effec107160 con 0x7effec0601e0 2024-02-19T22:08:45.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.420+0000 7effdaffd700 1 -- 172.21.15.176:0/3665010026 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7effdc0263a0 con 0x7effec0601e0 2024-02-19T22:08:45.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.420+0000 7effdaffd700 1 --2- 172.21.15.176:0/3665010026 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7effd402ff10 0x7effd40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:08:45.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.420+0000 7effdaffd700 1 -- 172.21.15.176:0/3665010026 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7effdc041180 con 0x7effec0601e0 2024-02-19T22:08:45.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.420+0000 7effeb59e700 1 -- 172.21.15.176:0/3665010026 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effec045ad0 con 0x7effec0601e0 2024-02-19T22:08:45.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.420+0000 7effe9d9b700 1 --2- 172.21.15.176:0/3665010026 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7effd402ff10 0x7effd40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:08:45.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.420+0000 7effe9d9b700 1 --2- 172.21.15.176:0/3665010026 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7effd402ff10 0x7effd40323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7effe00096a0 tx=0x7effe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:08:45.425 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.424+0000 7effdaffd700 1 -- 172.21.15.176:0/3665010026 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effdc012920 con 0x7effec0601e0 2024-02-19T22:08:45.585 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.584+0000 7effeb59e700 1 -- 172.21.15.176:0/3665010026 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7effec0fbf00 con 0x7effec0601e0 2024-02-19T22:08:45.588 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.587+0000 7effdaffd700 1 -- 172.21.15.176:0/3665010026 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7effdc012330 con 0x7effec0601e0 2024-02-19T22:08:45.590 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.589+0000 7effeb59e700 1 -- 172.21.15.176:0/3665010026 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7effd402ff10 msgr2=0x7effd40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:45.590 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.589+0000 7effeb59e700 1 --2- 172.21.15.176:0/3665010026 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7effd402ff10 0x7effd40323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7effe00096a0 tx=0x7effe0006b40).stop 2024-02-19T22:08:45.590 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.589+0000 7effeb59e700 1 -- 172.21.15.176:0/3665010026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effec0601e0 msgr2=0x7effec1065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:08:45.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.589+0000 7effeb59e700 1 --2- 172.21.15.176:0/3665010026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effec0601e0 0x7effec1065e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7effdc005010 tx=0x7effdc00b040).stop 2024-02-19T22:08:45.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.589+0000 7effeb59e700 1 -- 172.21.15.176:0/3665010026 shutdown_connections 2024-02-19T22:08:45.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.589+0000 7effeb59e700 1 --2- 172.21.15.176:0/3665010026 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7effd402ff10 0x7effd40323d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:45.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.589+0000 7effeb59e700 1 --2- 172.21.15.176:0/3665010026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7effec0601e0 0x7effec1065e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:08:45.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.589+0000 7effeb59e700 1 -- 172.21.15.176:0/3665010026 >> 172.21.15.176:0/3665010026 conn(0x7effec0f09a0 msgr2=0x7effec0f13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:08:45.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.590+0000 7effeb59e700 1 -- 172.21.15.176:0/3665010026 shutdown_connections 2024-02-19T22:08:45.591 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:08:45.590+0000 7effeb59e700 1 -- 172.21.15.176:0/3665010026 wait complete. 2024-02-19T22:08:46.304 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:46 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:45.216080+0000 mon.smithi176 (mon.0) 112 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:46.304 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:46 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:45.492513+0000 mon.smithi176 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:08:46.304 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:46 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:08:45.498650+0000 mgr.smithi176.faoogw (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi176 on smithi176 2024-02-19T22:08:46.304 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:46 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:08:45.589150+0000 mon.smithi176 (mon.0) 114 : audit [INF] from='client.? 172.21.15.176:0/3665010026' entity='client.admin' 2024-02-19T22:08:46.322 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi184 2024-02-19T22:08:46.322 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-02-19T22:08:46.323 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/etc/ceph/ceph.conf 2024-02-19T22:08:46.344 DEBUG:teuthology.orchestra.run.smithi184:> set -ex 2024-02-19T22:08:46.344 DEBUG:teuthology.orchestra.run.smithi184:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T22:08:46.406 INFO:tasks.cephadm:Adding host smithi184 to orchestrator... 2024-02-19T22:08:46.406 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph orch host add smithi184 2024-02-19T22:08:57.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:56 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:55.830950+0000 mgr.smithi176.faoogw (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:08:59.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:08:58 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:57.831198+0000 mgr.smithi176.faoogw (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:02.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:08:59.831476+0000 mgr.smithi176.faoogw (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:04.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:01.831776+0000 mgr.smithi176.faoogw (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:06.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:03.832140+0000 mgr.smithi176.faoogw (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:08.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:05.832425+0000 mgr.smithi176.faoogw (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:08.441 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.438+0000 7f9746d36700 1 -- 172.21.15.184:0/414673706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97400f6510 msgr2=0x7f97400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.438+0000 7f9746d36700 1 --2- 172.21.15.184:0/414673706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97400f6510 0x7f97400f6930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f9734009fb0 tx=0x7f9734009b30).stop 2024-02-19T22:09:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.438+0000 7f9746d36700 1 -- 172.21.15.184:0/414673706 shutdown_connections 2024-02-19T22:09:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.438+0000 7f9746d36700 1 --2- 172.21.15.184:0/414673706 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97400f6510 0x7f97400f6930 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.438+0000 7f9746d36700 1 -- 172.21.15.184:0/414673706 >> 172.21.15.184:0/414673706 conn(0x7f97400f1c70 msgr2=0x7f97400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.438+0000 7f9746d36700 1 -- 172.21.15.184:0/414673706 shutdown_connections 2024-02-19T22:09:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.438+0000 7f9746d36700 1 -- 172.21.15.184:0/414673706 wait complete. 2024-02-19T22:09:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.439+0000 7f9746d36700 1 Processor -- start 2024-02-19T22:09:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.439+0000 7f9746d36700 1 -- start start 2024-02-19T22:09:08.442 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.439+0000 7f9746d36700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97400f6510 0x7f974006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.439+0000 7f9746d36700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9740069e20 con 0x7f97400f6510 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.439+0000 7f9744ad2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97400f6510 0x7f974006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.439+0000 7f9744ad2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97400f6510 0x7f974006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:42396/0 (socket says 172.21.15.184:42396) 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.439+0000 7f9744ad2700 1 -- 172.21.15.184:0/3966642773 learned_addr learned my addr 172.21.15.184:0/3966642773 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.439+0000 7f9744ad2700 1 -- 172.21.15.184:0/3966642773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9734005040 con 0x7f97400f6510 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.439+0000 7f9744ad2700 1 --2- 172.21.15.184:0/3966642773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97400f6510 0x7f974006b9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f97340089f0 tx=0x7f973400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.440+0000 7f973d7fa700 1 -- 172.21.15.184:0/3966642773 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9734009320 con 0x7f97400f6510 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.440+0000 7f9746d36700 1 -- 172.21.15.184:0/3966642773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9740069fc0 con 0x7f97400f6510 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.440+0000 7f973d7fa700 1 -- 172.21.15.184:0/3966642773 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9734013380 con 0x7f97400f6510 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.440+0000 7f973d7fa700 1 -- 172.21.15.184:0/3966642773 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f973401ed80 con 0x7f97400f6510 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.440+0000 7f9746d36700 1 -- 172.21.15.184:0/3966642773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974006a450 con 0x7f97400f6510 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.440+0000 7f973d7fa700 1 -- 172.21.15.184:0/3966642773 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f973401e420 con 0x7f97400f6510 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.440+0000 7f9746d36700 1 -- 172.21.15.184:0/3966642773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9740059100 con 0x7f97400f6510 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.441+0000 7f973d7fa700 1 --2- 172.21.15.184:0/3966642773 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f973002ff60 0x7f9730032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:08.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.441+0000 7f973d7fa700 1 -- 172.21.15.184:0/3966642773 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f97340410b0 con 0x7f97400f6510 2024-02-19T22:09:08.444 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.441+0000 7f973ffff700 1 --2- 172.21.15.184:0/3966642773 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f973002ff60 0x7f9730032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:08.444 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.441+0000 7f973ffff700 1 --2- 172.21.15.184:0/3966642773 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f973002ff60 0x7f9730032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f972c0096a0 tx=0x7f972c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:08.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.445+0000 7f973d7fa700 1 -- 172.21.15.184:0/3966642773 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f973401cb10 con 0x7f97400f6510 2024-02-19T22:09:08.621 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:08.620+0000 7f9746d36700 1 -- 172.21.15.184:0/3966642773 --> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi184", "target": ["mon-mgr", ""]}) v1 -- 0x7f974006aca0 con 0x7f973002ff60 2024-02-19T22:09:09.212 INFO:teuthology.orchestra.run.smithi184.stdout:Added host 'smithi184' 2024-02-19T22:09:09.213 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:09.211+0000 7f973d7fa700 1 -- 172.21.15.184:0/3966642773 <== mgr.14162 v2:172.21.15.176:6800/3746885080 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f974006aca0 con 0x7f973002ff60 2024-02-19T22:09:09.214 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:09.213+0000 7f9746d36700 1 -- 172.21.15.184:0/3966642773 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f973002ff60 msgr2=0x7f9730032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:09.214 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:09.213+0000 7f9746d36700 1 --2- 172.21.15.184:0/3966642773 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f973002ff60 0x7f9730032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f972c0096a0 tx=0x7f972c006b40).stop 2024-02-19T22:09:09.214 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:09.213+0000 7f9746d36700 1 -- 172.21.15.184:0/3966642773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97400f6510 msgr2=0x7f974006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:09.214 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:09.213+0000 7f9746d36700 1 --2- 172.21.15.184:0/3966642773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97400f6510 0x7f974006b9a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f97340089f0 tx=0x7f973400b040).stop 2024-02-19T22:09:09.214 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:09.213+0000 7f9746d36700 1 -- 172.21.15.184:0/3966642773 shutdown_connections 2024-02-19T22:09:09.214 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:09.213+0000 7f9746d36700 1 --2- 172.21.15.184:0/3966642773 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f973002ff60 0x7f9730032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:09.214 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:09.213+0000 7f9746d36700 1 --2- 172.21.15.184:0/3966642773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f97400f6510 0x7f974006b9a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:09.214 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:09.213+0000 7f9746d36700 1 -- 172.21.15.184:0/3966642773 >> 172.21.15.184:0/3966642773 conn(0x7f97400f1c70 msgr2=0x7f97400f26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:09.214 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:09.213+0000 7f9746d36700 1 -- 172.21.15.184:0/3966642773 shutdown_connections 2024-02-19T22:09:09.214 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:09.213+0000 7f9746d36700 1 -- 172.21.15.184:0/3966642773 wait complete. 2024-02-19T22:09:09.852 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-19T22:09:10.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:07.832758+0000 mgr.smithi176.faoogw (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:10.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:08.622773+0000 mgr.smithi176.faoogw (mgr.14162) 13 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi184", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:09:10.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:09.211653+0000 mon.smithi176 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:09:11.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:10 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:09:09.212037+0000 mgr.smithi176.faoogw (mgr.14162) 14 : cephadm [INF] Added host smithi184 2024-02-19T22:09:11.582 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.580+0000 7f715f6d4700 1 -- 172.21.15.184:0/2388974555 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71580f6510 msgr2=0x7f71580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:11.583 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.580+0000 7f715f6d4700 1 --2- 172.21.15.184:0/2388974555 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71580f6510 0x7f71580f6930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f7148009fb0 tx=0x7f7148009b30).stop 2024-02-19T22:09:11.583 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.580+0000 7f715f6d4700 1 -- 172.21.15.184:0/2388974555 shutdown_connections 2024-02-19T22:09:11.583 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.580+0000 7f715f6d4700 1 --2- 172.21.15.184:0/2388974555 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71580f6510 0x7f71580f6930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:11.583 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.580+0000 7f715f6d4700 1 -- 172.21.15.184:0/2388974555 >> 172.21.15.184:0/2388974555 conn(0x7f71580f1c70 msgr2=0x7f71580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:11.583 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.580+0000 7f715f6d4700 1 -- 172.21.15.184:0/2388974555 shutdown_connections 2024-02-19T22:09:11.583 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.580+0000 7f715f6d4700 1 -- 172.21.15.184:0/2388974555 wait complete. 2024-02-19T22:09:11.583 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.581+0000 7f715f6d4700 1 Processor -- start 2024-02-19T22:09:11.583 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.581+0000 7f715f6d4700 1 -- start start 2024-02-19T22:09:11.583 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.581+0000 7f715f6d4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71580f6510 0x7f71581067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:11.583 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.581+0000 7f715f6d4700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7158106cd0 con 0x7f71580f6510 2024-02-19T22:09:11.584 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.581+0000 7f715d470700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71580f6510 0x7f71581067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:11.584 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.581+0000 7f715d470700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71580f6510 0x7f71581067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:42420/0 (socket says 172.21.15.184:42420) 2024-02-19T22:09:11.584 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.581+0000 7f715d470700 1 -- 172.21.15.184:0/1793794137 learned_addr learned my addr 172.21.15.184:0/1793794137 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:09:11.584 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.582+0000 7f715d470700 1 -- 172.21.15.184:0/1793794137 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7148005040 con 0x7f71580f6510 2024-02-19T22:09:11.584 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.582+0000 7f715d470700 1 --2- 172.21.15.184:0/1793794137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71580f6510 0x7f71581067e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7148005010 tx=0x7f714800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:11.584 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.582+0000 7f714dffb700 1 -- 172.21.15.184:0/1793794137 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7148009320 con 0x7f71580f6510 2024-02-19T22:09:11.584 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.582+0000 7f715f6d4700 1 -- 172.21.15.184:0/1793794137 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7158106ed0 con 0x7f71580f6510 2024-02-19T22:09:11.584 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.582+0000 7f714dffb700 1 -- 172.21.15.184:0/1793794137 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7148013380 con 0x7f71580f6510 2024-02-19T22:09:11.584 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.582+0000 7f714dffb700 1 -- 172.21.15.184:0/1793794137 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7148014e50 con 0x7f71580f6510 2024-02-19T22:09:11.584 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.582+0000 7f715f6d4700 1 -- 172.21.15.184:0/1793794137 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7158107360 con 0x7f71580f6510 2024-02-19T22:09:11.585 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.583+0000 7f714dffb700 1 -- 172.21.15.184:0/1793794137 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71480095f0 con 0x7f71580f6510 2024-02-19T22:09:11.585 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.583+0000 7f715f6d4700 1 -- 172.21.15.184:0/1793794137 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7158045ad0 con 0x7f71580f6510 2024-02-19T22:09:11.585 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.583+0000 7f714dffb700 1 --2- 172.21.15.184:0/1793794137 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f714402ff10 0x7f71440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:11.585 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.583+0000 7f714dffb700 1 -- 172.21.15.184:0/1793794137 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7148040e90 con 0x7f71580f6510 2024-02-19T22:09:11.585 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.583+0000 7f715cc6f700 1 --2- 172.21.15.184:0/1793794137 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f714402ff10 0x7f71440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:11.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.584+0000 7f715cc6f700 1 --2- 172.21.15.184:0/1793794137 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f714402ff10 0x7f71440323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f71540096a0 tx=0x7f7154006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:11.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.588+0000 7f714dffb700 1 -- 172.21.15.184:0/1793794137 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7148014420 con 0x7f71580f6510 2024-02-19T22:09:11.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.760+0000 7f715f6d4700 1 -- 172.21.15.184:0/1793794137 --> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f71580f2e20 con 0x7f714402ff10 2024-02-19T22:09:11.763 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.762+0000 7f714dffb700 1 -- 172.21.15.184:0/1793794137 <== mgr.14162 v2:172.21.15.176:6800/3746885080 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f71580f2e20 con 0x7f714402ff10 2024-02-19T22:09:11.763 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:09:11.763 INFO:teuthology.orchestra.run.smithi184.stdout:[{"addr": "172.21.15.176", "hostname": "smithi176", "labels": [], "status": ""}, {"addr": "smithi184", "hostname": "smithi184", "labels": [], "status": ""}] 2024-02-19T22:09:11.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.764+0000 7f715f6d4700 1 -- 172.21.15.184:0/1793794137 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f714402ff10 msgr2=0x7f71440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:11.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.764+0000 7f715f6d4700 1 --2- 172.21.15.184:0/1793794137 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f714402ff10 0x7f71440323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f71540096a0 tx=0x7f7154006b40).stop 2024-02-19T22:09:11.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.764+0000 7f715f6d4700 1 -- 172.21.15.184:0/1793794137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71580f6510 msgr2=0x7f71581067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:11.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.764+0000 7f715f6d4700 1 --2- 172.21.15.184:0/1793794137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71580f6510 0x7f71581067e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f7148005010 tx=0x7f714800b040).stop 2024-02-19T22:09:11.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.764+0000 7f715f6d4700 1 -- 172.21.15.184:0/1793794137 shutdown_connections 2024-02-19T22:09:11.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.764+0000 7f715f6d4700 1 --2- 172.21.15.184:0/1793794137 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f714402ff10 0x7f71440323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:11.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.764+0000 7f715f6d4700 1 --2- 172.21.15.184:0/1793794137 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71580f6510 0x7f71581067e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:11.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.764+0000 7f715f6d4700 1 -- 172.21.15.184:0/1793794137 >> 172.21.15.184:0/1793794137 conn(0x7f71580f1c70 msgr2=0x7f71580f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:11.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.764+0000 7f715f6d4700 1 -- 172.21.15.184:0/1793794137 shutdown_connections 2024-02-19T22:09:11.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:11.764+0000 7f715f6d4700 1 -- 172.21.15.184:0/1793794137 wait complete. 2024-02-19T22:09:12.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:09.833168+0000 mgr.smithi176.faoogw (mgr.14162) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:12.287 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-19T22:09:12.288 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-19T22:09:13.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:12 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:11.763032+0000 mgr.smithi176.faoogw (mgr.14162) 16 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T22:09:13.968 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:13 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:11.833419+0000 mgr.smithi176.faoogw (mgr.14162) 17 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:14.188 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.184+0000 7f87cfa43700 1 -- 172.21.15.176:0/76471264 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87c80f5410 msgr2=0x7f87c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:14.188 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.184+0000 7f87cfa43700 1 --2- 172.21.15.176:0/76471264 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87c80f5410 0x7f87c80f5830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f87b8009fb0 tx=0x7f87b8009b30).stop 2024-02-19T22:09:14.188 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.184+0000 7f87cfa43700 1 -- 172.21.15.176:0/76471264 shutdown_connections 2024-02-19T22:09:14.188 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.184+0000 7f87cfa43700 1 --2- 172.21.15.176:0/76471264 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87c80f5410 0x7f87c80f5830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:14.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.184+0000 7f87cfa43700 1 -- 172.21.15.176:0/76471264 >> 172.21.15.176:0/76471264 conn(0x7f87c80f0b70 msgr2=0x7f87c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:14.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.184+0000 7f87cfa43700 1 -- 172.21.15.176:0/76471264 shutdown_connections 2024-02-19T22:09:14.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.184+0000 7f87cfa43700 1 -- 172.21.15.176:0/76471264 wait complete. 2024-02-19T22:09:14.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.185+0000 7f87cfa43700 1 Processor -- start 2024-02-19T22:09:14.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.185+0000 7f87cfa43700 1 -- start start 2024-02-19T22:09:14.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.185+0000 7f87cfa43700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87c80f5410 0x7f87c81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:14.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.185+0000 7f87cfa43700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c8106cd0 con 0x7f87c80f5410 2024-02-19T22:09:14.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.185+0000 7f87cd7df700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87c80f5410 0x7f87c81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:14.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.185+0000 7f87cd7df700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87c80f5410 0x7f87c81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:56742/0 (socket says 172.21.15.176:56742) 2024-02-19T22:09:14.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.185+0000 7f87cd7df700 1 -- 172.21.15.176:0/2781753334 learned_addr learned my addr 172.21.15.176:0/2781753334 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-02-19T22:09:14.189 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.185+0000 7f87cd7df700 1 -- 172.21.15.176:0/2781753334 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b8005040 con 0x7f87c80f5410 2024-02-19T22:09:14.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.185+0000 7f87cd7df700 1 --2- 172.21.15.176:0/2781753334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87c80f5410 0x7f87c81067e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f87b8005010 tx=0x7f87b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:14.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.186+0000 7f87bdffb700 1 -- 172.21.15.176:0/2781753334 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87b8009320 con 0x7f87c80f5410 2024-02-19T22:09:14.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.186+0000 7f87bdffb700 1 -- 172.21.15.176:0/2781753334 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87b8013380 con 0x7f87c80f5410 2024-02-19T22:09:14.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.186+0000 7f87cfa43700 1 -- 172.21.15.176:0/2781753334 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c8106ed0 con 0x7f87c80f5410 2024-02-19T22:09:14.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.186+0000 7f87bdffb700 1 -- 172.21.15.176:0/2781753334 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87b8014e50 con 0x7f87c80f5410 2024-02-19T22:09:14.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.186+0000 7f87cfa43700 1 -- 172.21.15.176:0/2781753334 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c8107360 con 0x7f87c80f5410 2024-02-19T22:09:14.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.187+0000 7f87bdffb700 1 -- 172.21.15.176:0/2781753334 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87b80263a0 con 0x7f87c80f5410 2024-02-19T22:09:14.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.187+0000 7f87bdffb700 1 --2- 172.21.15.176:0/2781753334 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f87b402ff10 0x7f87b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:14.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.187+0000 7f87bdffb700 1 -- 172.21.15.176:0/2781753334 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87b8040e70 con 0x7f87c80f5410 2024-02-19T22:09:14.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.187+0000 7f87cfa43700 1 -- 172.21.15.176:0/2781753334 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c8045ad0 con 0x7f87c80f5410 2024-02-19T22:09:14.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.187+0000 7f87ccfde700 1 --2- 172.21.15.176:0/2781753334 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f87b402ff10 0x7f87b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:14.190 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.187+0000 7f87ccfde700 1 --2- 172.21.15.176:0/2781753334 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f87b402ff10 0x7f87b40323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f87c40096a0 tx=0x7f87c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:14.192 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.191+0000 7f87bdffb700 1 -- 172.21.15.176:0/2781753334 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87b8012920 con 0x7f87c80f5410 2024-02-19T22:09:14.347 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.345+0000 7f87cfa43700 1 -- 172.21.15.176:0/2781753334 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f87c81080a0 con 0x7f87c80f5410 2024-02-19T22:09:14.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.746+0000 7f87bdffb700 1 -- 172.21.15.176:0/2781753334 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f87b800f340 con 0x7f87c80f5410 2024-02-19T22:09:14.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.748+0000 7f87cfa43700 1 -- 172.21.15.176:0/2781753334 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f87b402ff10 msgr2=0x7f87b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:14.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.748+0000 7f87cfa43700 1 --2- 172.21.15.176:0/2781753334 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f87b402ff10 0x7f87b40323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f87c40096a0 tx=0x7f87c4006b40).stop 2024-02-19T22:09:14.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.748+0000 7f87cfa43700 1 -- 172.21.15.176:0/2781753334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87c80f5410 msgr2=0x7f87c81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:14.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.748+0000 7f87cfa43700 1 --2- 172.21.15.176:0/2781753334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87c80f5410 0x7f87c81067e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f87b8005010 tx=0x7f87b800b040).stop 2024-02-19T22:09:14.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.748+0000 7f87cfa43700 1 -- 172.21.15.176:0/2781753334 shutdown_connections 2024-02-19T22:09:14.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.748+0000 7f87cfa43700 1 --2- 172.21.15.176:0/2781753334 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f87b402ff10 0x7f87b40323d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:14.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.748+0000 7f87cfa43700 1 --2- 172.21.15.176:0/2781753334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87c80f5410 0x7f87c81067e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:14.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.748+0000 7f87cfa43700 1 -- 172.21.15.176:0/2781753334 >> 172.21.15.176:0/2781753334 conn(0x7f87c80f0b70 msgr2=0x7f87c80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:14.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.748+0000 7f87cfa43700 1 -- 172.21.15.176:0/2781753334 shutdown_connections 2024-02-19T22:09:14.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-02-19T22:09:14.748+0000 7f87cfa43700 1 -- 172.21.15.176:0/2781753334 wait complete. 2024-02-19T22:09:14.750 INFO:teuthology.orchestra.run.smithi176.stderr:adjusted tunables profile to default 2024-02-19T22:09:15.100 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:14 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:14.347768+0000 mon.smithi176 (mon.0) 116 : audit [INF] from='client.? 172.21.15.176:0/2781753334' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-19T22:09:15.452 INFO:tasks.cephadm:Adding mon.smithi176 on smithi176 2024-02-19T22:09:15.452 INFO:tasks.cephadm:Adding mon.smithi184 on smithi184 2024-02-19T22:09:15.452 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi176:172.21.15.176=smithi176;smithi184:172.21.15.184=smithi184' 2024-02-19T22:09:16.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:13.833733+0000 mgr.smithi176.faoogw (mgr.14162) 18 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:16.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:14.747555+0000 mon.smithi176 (mon.0) 117 : audit [INF] from='client.? 172.21.15.176:0/2781753334' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-19T22:09:16.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:14.747596+0000 mon.smithi176 (mon.0) 118 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-19T22:09:17.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.247+0000 7fa94ad7e700 1 -- 172.21.15.184:0/4216004871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9440f5410 msgr2=0x7fa9440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:17.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.247+0000 7fa94ad7e700 1 --2- 172.21.15.184:0/4216004871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9440f5410 0x7fa9440f5830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fa938009fb0 tx=0x7fa938009b30).stop 2024-02-19T22:09:17.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.247+0000 7fa94ad7e700 1 -- 172.21.15.184:0/4216004871 shutdown_connections 2024-02-19T22:09:17.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.247+0000 7fa94ad7e700 1 --2- 172.21.15.184:0/4216004871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9440f5410 0x7fa9440f5830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:17.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.247+0000 7fa94ad7e700 1 -- 172.21.15.184:0/4216004871 >> 172.21.15.184:0/4216004871 conn(0x7fa9440f0b70 msgr2=0x7fa9440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:17.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.247+0000 7fa94ad7e700 1 -- 172.21.15.184:0/4216004871 shutdown_connections 2024-02-19T22:09:17.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.247+0000 7fa94ad7e700 1 -- 172.21.15.184:0/4216004871 wait complete. 2024-02-19T22:09:17.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.248+0000 7fa94ad7e700 1 Processor -- start 2024-02-19T22:09:17.249 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.248+0000 7fa94ad7e700 1 -- start start 2024-02-19T22:09:17.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.248+0000 7fa94ad7e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9440f5410 0x7fa9441094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.248+0000 7fa94ad7e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa944060240 con 0x7fa9440f5410 2024-02-19T22:09:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.248+0000 7fa948b1a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9440f5410 0x7fa9441094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.248+0000 7fa948b1a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9440f5410 0x7fa9441094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:35908/0 (socket says 172.21.15.184:35908) 2024-02-19T22:09:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.248+0000 7fa948b1a700 1 -- 172.21.15.184:0/1847953998 learned_addr learned my addr 172.21.15.184:0/1847953998 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:09:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.249+0000 7fa948b1a700 1 -- 172.21.15.184:0/1847953998 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa938005040 con 0x7fa9440f5410 2024-02-19T22:09:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.249+0000 7fa948b1a700 1 --2- 172.21.15.184:0/1847953998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9440f5410 0x7fa9441094e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fa938005010 tx=0x7fa93800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.249+0000 7fa9417fa700 1 -- 172.21.15.184:0/1847953998 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa938009320 con 0x7fa9440f5410 2024-02-19T22:09:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.249+0000 7fa94ad7e700 1 -- 172.21.15.184:0/1847953998 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa944060440 con 0x7fa9440f5410 2024-02-19T22:09:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.249+0000 7fa9417fa700 1 -- 172.21.15.184:0/1847953998 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa938013380 con 0x7fa9440f5410 2024-02-19T22:09:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.249+0000 7fa9417fa700 1 -- 172.21.15.184:0/1847953998 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa938014e50 con 0x7fa9440f5410 2024-02-19T22:09:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.249+0000 7fa94ad7e700 1 -- 172.21.15.184:0/1847953998 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa944109b90 con 0x7fa9440f5410 2024-02-19T22:09:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.250+0000 7fa9417fa700 1 -- 172.21.15.184:0/1847953998 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9380095f0 con 0x7fa9440f5410 2024-02-19T22:09:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.250+0000 7fa94ad7e700 1 -- 172.21.15.184:0/1847953998 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa944045ad0 con 0x7fa9440f5410 2024-02-19T22:09:17.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.250+0000 7fa9417fa700 1 --2- 172.21.15.184:0/1847953998 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa93402ff10 0x7fa9340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:17.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.250+0000 7fa9417fa700 1 -- 172.21.15.184:0/1847953998 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa938040e90 con 0x7fa9440f5410 2024-02-19T22:09:17.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.250+0000 7fa943fff700 1 --2- 172.21.15.184:0/1847953998 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa93402ff10 0x7fa9340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:17.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.250+0000 7fa943fff700 1 --2- 172.21.15.184:0/1847953998 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa93402ff10 0x7fa9340323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa9300096a0 tx=0x7fa930006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:17.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.255+0000 7fa9417fa700 1 -- 172.21.15.184:0/1847953998 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa938014420 con 0x7fa9440f5410 2024-02-19T22:09:17.429 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.429+0000 7fa94ad7e700 1 -- 172.21.15.184:0/1847953998 --> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi176:172.21.15.176=smithi176;smithi184:172.21.15.184=smithi184", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9440f1dd0 con 0x7fa93402ff10 2024-02-19T22:09:17.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.434+0000 7fa9417fa700 1 -- 172.21.15.184:0/1847953998 <== mgr.14162 v2:172.21.15.176:6800/3746885080 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa9440f1dd0 con 0x7fa93402ff10 2024-02-19T22:09:17.435 INFO:teuthology.orchestra.run.smithi184.stdout:Scheduled mon update... 2024-02-19T22:09:17.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.436+0000 7fa94ad7e700 1 -- 172.21.15.184:0/1847953998 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa93402ff10 msgr2=0x7fa9340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:17.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.436+0000 7fa94ad7e700 1 --2- 172.21.15.184:0/1847953998 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa93402ff10 0x7fa9340323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa9300096a0 tx=0x7fa930006b40).stop 2024-02-19T22:09:17.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.436+0000 7fa94ad7e700 1 -- 172.21.15.184:0/1847953998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9440f5410 msgr2=0x7fa9441094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:17.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.436+0000 7fa94ad7e700 1 --2- 172.21.15.184:0/1847953998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9440f5410 0x7fa9441094e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fa938005010 tx=0x7fa93800b040).stop 2024-02-19T22:09:17.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.436+0000 7fa94ad7e700 1 -- 172.21.15.184:0/1847953998 shutdown_connections 2024-02-19T22:09:17.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.436+0000 7fa94ad7e700 1 --2- 172.21.15.184:0/1847953998 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa93402ff10 0x7fa9340323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:17.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.436+0000 7fa94ad7e700 1 --2- 172.21.15.184:0/1847953998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa9440f5410 0x7fa9441094e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:17.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.436+0000 7fa94ad7e700 1 -- 172.21.15.184:0/1847953998 >> 172.21.15.184:0/1847953998 conn(0x7fa9440f0b70 msgr2=0x7fa9440f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:17.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.436+0000 7fa94ad7e700 1 -- 172.21.15.184:0/1847953998 shutdown_connections 2024-02-19T22:09:17.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:17.436+0000 7fa94ad7e700 1 -- 172.21.15.184:0/1847953998 wait complete. 2024-02-19T22:09:18.011 DEBUG:teuthology.orchestra.run.smithi184:mon.smithi184> sudo journalctl -f -n 0 -u ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi184.service 2024-02-19T22:09:18.013 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:09:18.013 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:09:18.041 INFO:journalctl@ceph.mon.smithi184.smithi184.stdout:-- Logs begin at Mon 2024-02-19 21:56:57 UTC. -- 2024-02-19T22:09:18.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:17 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:15.833999+0000 mgr.smithi176.faoogw (mgr.14162) 19 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:18.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:17 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:17.434914+0000 mon.smithi176 (mon.0) 119 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:09:19.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:18 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:17.431124+0000 mgr.smithi176.faoogw (mgr.14162) 20 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi176:172.21.15.176=smithi176;smithi184:172.21.15.184=smithi184", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:09:19.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:18 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:09:17.432526+0000 mgr.smithi176.faoogw (mgr.14162) 21 : cephadm [INF] Saving service mon spec with placement smithi176:172.21.15.176=smithi176;smithi184:172.21.15.184=smithi184;count:2 2024-02-19T22:09:19.786 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.784+0000 7ff86294d700 1 -- 172.21.15.184:0/448892193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff85c0601e0 msgr2=0x7ff85c0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:19.786 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.784+0000 7ff86294d700 1 --2- 172.21.15.184:0/448892193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff85c0601e0 0x7ff85c0f9190 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7ff844009fb0 tx=0x7ff844009b30).stop 2024-02-19T22:09:19.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.784+0000 7ff86294d700 1 -- 172.21.15.184:0/448892193 shutdown_connections 2024-02-19T22:09:19.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.784+0000 7ff86294d700 1 --2- 172.21.15.184:0/448892193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff85c0601e0 0x7ff85c0f9190 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:19.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.784+0000 7ff86294d700 1 -- 172.21.15.184:0/448892193 >> 172.21.15.184:0/448892193 conn(0x7ff85c0f0b70 msgr2=0x7ff85c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:19.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.784+0000 7ff86294d700 1 -- 172.21.15.184:0/448892193 shutdown_connections 2024-02-19T22:09:19.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.784+0000 7ff86294d700 1 -- 172.21.15.184:0/448892193 wait complete. 2024-02-19T22:09:19.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.785+0000 7ff86294d700 1 Processor -- start 2024-02-19T22:09:19.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.785+0000 7ff86294d700 1 -- start start 2024-02-19T22:09:19.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.785+0000 7ff86294d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff85c0601e0 0x7ff85c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:19.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.785+0000 7ff86294d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c106cc0 con 0x7ff85c0601e0 2024-02-19T22:09:19.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.785+0000 7ff85bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff85c0601e0 0x7ff85c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:19.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.785+0000 7ff85bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff85c0601e0 0x7ff85c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:35930/0 (socket says 172.21.15.184:35930) 2024-02-19T22:09:19.787 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.785+0000 7ff85bfff700 1 -- 172.21.15.184:0/1837782307 learned_addr learned my addr 172.21.15.184:0/1837782307 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:09:19.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.785+0000 7ff85bfff700 1 -- 172.21.15.184:0/1837782307 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff844005040 con 0x7ff85c0601e0 2024-02-19T22:09:19.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.786+0000 7ff85bfff700 1 --2- 172.21.15.184:0/1837782307 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff85c0601e0 0x7ff85c1067d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7ff844007a10 tx=0x7ff844009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:19.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.786+0000 7ff858ff9700 1 -- 172.21.15.184:0/1837782307 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff844018070 con 0x7ff85c0601e0 2024-02-19T22:09:19.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.786+0000 7ff86294d700 1 -- 172.21.15.184:0/1837782307 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85c106ec0 con 0x7ff85c0601e0 2024-02-19T22:09:19.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.786+0000 7ff858ff9700 1 -- 172.21.15.184:0/1837782307 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff844013410 con 0x7ff85c0601e0 2024-02-19T22:09:19.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.786+0000 7ff858ff9700 1 -- 172.21.15.184:0/1837782307 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff84401c3c0 con 0x7ff85c0601e0 2024-02-19T22:09:19.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.786+0000 7ff86294d700 1 -- 172.21.15.184:0/1837782307 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85c107350 con 0x7ff85c0601e0 2024-02-19T22:09:19.789 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.786+0000 7ff858ff9700 1 -- 172.21.15.184:0/1837782307 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff84401d420 con 0x7ff85c0601e0 2024-02-19T22:09:19.789 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.787+0000 7ff86294d700 1 -- 172.21.15.184:0/1837782307 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85c045ad0 con 0x7ff85c0601e0 2024-02-19T22:09:19.789 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.787+0000 7ff858ff9700 1 --2- 172.21.15.184:0/1837782307 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff84802ff60 0x7ff848032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:19.789 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.787+0000 7ff858ff9700 1 -- 172.21.15.184:0/1837782307 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff844041570 con 0x7ff85c0601e0 2024-02-19T22:09:19.789 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.787+0000 7ff85b7fe700 1 --2- 172.21.15.184:0/1837782307 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff84802ff60 0x7ff848032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:19.789 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.787+0000 7ff85b7fe700 1 --2- 172.21.15.184:0/1837782307 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff84802ff60 0x7ff848032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff84c0096a0 tx=0x7ff84c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:19.792 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.791+0000 7ff858ff9700 1 -- 172.21.15.184:0/1837782307 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff84401d820 con 0x7ff85c0601e0 2024-02-19T22:09:19.995 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.995+0000 7ff86294d700 1 -- 172.21.15.184:0/1837782307 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff85c1081a0 con 0x7ff85c0601e0 2024-02-19T22:09:19.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.995+0000 7ff858ff9700 1 -- 172.21.15.184:0/1837782307 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff84401b370 con 0x7ff85c0601e0 2024-02-19T22:09:19.996 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:09:19.996 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:09:19.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.997+0000 7ff86294d700 1 -- 172.21.15.184:0/1837782307 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff84802ff60 msgr2=0x7ff848032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:19.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.997+0000 7ff86294d700 1 --2- 172.21.15.184:0/1837782307 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff84802ff60 0x7ff848032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff84c0096a0 tx=0x7ff84c006b40).stop 2024-02-19T22:09:19.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.997+0000 7ff86294d700 1 -- 172.21.15.184:0/1837782307 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff85c0601e0 msgr2=0x7ff85c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:19.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.997+0000 7ff86294d700 1 --2- 172.21.15.184:0/1837782307 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff85c0601e0 0x7ff85c1067d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7ff844007a10 tx=0x7ff844009b30).stop 2024-02-19T22:09:19.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.997+0000 7ff86294d700 1 -- 172.21.15.184:0/1837782307 shutdown_connections 2024-02-19T22:09:19.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.997+0000 7ff86294d700 1 --2- 172.21.15.184:0/1837782307 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff84802ff60 0x7ff848032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:19.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.997+0000 7ff86294d700 1 --2- 172.21.15.184:0/1837782307 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff85c0601e0 0x7ff85c1067d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:19.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.997+0000 7ff86294d700 1 -- 172.21.15.184:0/1837782307 >> 172.21.15.184:0/1837782307 conn(0x7ff85c0f0b70 msgr2=0x7ff85c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:19.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.997+0000 7ff86294d700 1 -- 172.21.15.184:0/1837782307 shutdown_connections 2024-02-19T22:09:19.998 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:19.997+0000 7ff86294d700 1 -- 172.21.15.184:0/1837782307 wait complete. 2024-02-19T22:09:19.999 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:09:20.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:17.834248+0000 mgr.smithi176.faoogw (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:21.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:20 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:19.996651+0000 mon.smithi176 (mon.0) 120 : audit [DBG] from='client.? 172.21.15.184:0/1837782307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:09:21.604 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:09:21.605 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:09:22.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:19.834646+0000 mgr.smithi176.faoogw (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:23.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.194+0000 7f1442486700 1 -- 172.21.15.184:0/1495001094 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f143c0601e0 msgr2=0x7f143c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:23.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.194+0000 7f1442486700 1 --2- 172.21.15.184:0/1495001094 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f143c0601e0 0x7f143c0f8f20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f1438009fb0 tx=0x7f1438009b30).stop 2024-02-19T22:09:23.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.194+0000 7f1442486700 1 -- 172.21.15.184:0/1495001094 shutdown_connections 2024-02-19T22:09:23.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.194+0000 7f1442486700 1 --2- 172.21.15.184:0/1495001094 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f143c0601e0 0x7f143c0f8f20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:23.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.194+0000 7f1442486700 1 -- 172.21.15.184:0/1495001094 >> 172.21.15.184:0/1495001094 conn(0x7f143c0f09a0 msgr2=0x7f143c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:23.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.194+0000 7f1442486700 1 -- 172.21.15.184:0/1495001094 shutdown_connections 2024-02-19T22:09:23.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.194+0000 7f1442486700 1 -- 172.21.15.184:0/1495001094 wait complete. 2024-02-19T22:09:23.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.194+0000 7f1442486700 1 Processor -- start 2024-02-19T22:09:23.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.194+0000 7f1442486700 1 -- start start 2024-02-19T22:09:23.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.194+0000 7f1442486700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f143c0601e0 0x7f143c106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:23.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.194+0000 7f1442486700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143c106af0 con 0x7f143c0601e0 2024-02-19T22:09:23.196 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.195+0000 7f1441484700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f143c0601e0 0x7f143c106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:23.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.195+0000 7f1441484700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f143c0601e0 0x7f143c106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:35952/0 (socket says 172.21.15.184:35952) 2024-02-19T22:09:23.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.195+0000 7f1441484700 1 -- 172.21.15.184:0/4119112240 learned_addr learned my addr 172.21.15.184:0/4119112240 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:09:23.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.195+0000 7f1441484700 1 -- 172.21.15.184:0/4119112240 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1438005040 con 0x7f143c0601e0 2024-02-19T22:09:23.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.195+0000 7f1441484700 1 --2- 172.21.15.184:0/4119112240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f143c0601e0 0x7f143c106600 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f1438005010 tx=0x7f143800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:23.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.195+0000 7f1431ffb700 1 -- 172.21.15.184:0/4119112240 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1438009320 con 0x7f143c0601e0 2024-02-19T22:09:23.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.195+0000 7f1442486700 1 -- 172.21.15.184:0/4119112240 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143c106cf0 con 0x7f143c0601e0 2024-02-19T22:09:23.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.195+0000 7f1431ffb700 1 -- 172.21.15.184:0/4119112240 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1438013380 con 0x7f143c0601e0 2024-02-19T22:09:23.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.195+0000 7f1431ffb700 1 -- 172.21.15.184:0/4119112240 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1438014e50 con 0x7f143c0601e0 2024-02-19T22:09:23.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.195+0000 7f1442486700 1 -- 172.21.15.184:0/4119112240 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f143c107180 con 0x7f143c0601e0 2024-02-19T22:09:23.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.196+0000 7f1442486700 1 -- 172.21.15.184:0/4119112240 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143c045ad0 con 0x7f143c0601e0 2024-02-19T22:09:23.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.196+0000 7f1431ffb700 1 -- 172.21.15.184:0/4119112240 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14380095f0 con 0x7f143c0601e0 2024-02-19T22:09:23.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.196+0000 7f1431ffb700 1 --2- 172.21.15.184:0/4119112240 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f142802ff10 0x7f14280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:23.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.196+0000 7f1431ffb700 1 -- 172.21.15.184:0/4119112240 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1438040e70 con 0x7f143c0601e0 2024-02-19T22:09:23.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.197+0000 7f1440c83700 1 --2- 172.21.15.184:0/4119112240 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f142802ff10 0x7f14280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:23.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.197+0000 7f1440c83700 1 --2- 172.21.15.184:0/4119112240 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f142802ff10 0x7f14280323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f142c0096a0 tx=0x7f142c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:23.201 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.201+0000 7f1431ffb700 1 -- 172.21.15.184:0/4119112240 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1438014420 con 0x7f143c0601e0 2024-02-19T22:09:23.405 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.404+0000 7f1442486700 1 -- 172.21.15.184:0/4119112240 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f143c107fd0 con 0x7f143c0601e0 2024-02-19T22:09:23.405 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.405+0000 7f1431ffb700 1 -- 172.21.15.184:0/4119112240 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1438010230 con 0x7f143c0601e0 2024-02-19T22:09:23.406 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:09:23.406 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:09:23.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.406+0000 7f1442486700 1 -- 172.21.15.184:0/4119112240 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f142802ff10 msgr2=0x7f14280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:23.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.406+0000 7f1442486700 1 --2- 172.21.15.184:0/4119112240 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f142802ff10 0x7f14280323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f142c0096a0 tx=0x7f142c006b40).stop 2024-02-19T22:09:23.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.407+0000 7f1442486700 1 -- 172.21.15.184:0/4119112240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f143c0601e0 msgr2=0x7f143c106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:23.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.407+0000 7f1442486700 1 --2- 172.21.15.184:0/4119112240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f143c0601e0 0x7f143c106600 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f1438005010 tx=0x7f143800b040).stop 2024-02-19T22:09:23.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.407+0000 7f1442486700 1 -- 172.21.15.184:0/4119112240 shutdown_connections 2024-02-19T22:09:23.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.407+0000 7f1442486700 1 --2- 172.21.15.184:0/4119112240 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f142802ff10 0x7f14280323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:23.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.407+0000 7f1442486700 1 --2- 172.21.15.184:0/4119112240 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f143c0601e0 0x7f143c106600 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:23.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.407+0000 7f1442486700 1 -- 172.21.15.184:0/4119112240 >> 172.21.15.184:0/4119112240 conn(0x7f143c0f09a0 msgr2=0x7f143c0f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:23.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.407+0000 7f1442486700 1 -- 172.21.15.184:0/4119112240 shutdown_connections 2024-02-19T22:09:23.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:23.407+0000 7f1442486700 1 -- 172.21.15.184:0/4119112240 wait complete. 2024-02-19T22:09:23.408 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:09:24.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:21.834858+0000 mgr.smithi176.faoogw (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:24.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:23.406136+0000 mon.smithi176 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.184:0/4119112240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:09:25.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:09:25.023 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:09:26.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:23.835210+0000 mgr.smithi176.faoogw (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:26.622 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.621+0000 7fec2293e700 1 -- 172.21.15.184:0/850382469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec1c0f5330 msgr2=0x7fec1c0f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:26.622 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.621+0000 7fec2293e700 1 --2- 172.21.15.184:0/850382469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec1c0f5330 0x7fec1c0f5750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fec0c009fb0 tx=0x7fec0c009b30).stop 2024-02-19T22:09:26.623 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.621+0000 7fec2293e700 1 -- 172.21.15.184:0/850382469 shutdown_connections 2024-02-19T22:09:26.623 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.621+0000 7fec2293e700 1 --2- 172.21.15.184:0/850382469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec1c0f5330 0x7fec1c0f5750 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:26.623 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.621+0000 7fec2293e700 1 -- 172.21.15.184:0/850382469 >> 172.21.15.184:0/850382469 conn(0x7fec1c0f0a90 msgr2=0x7fec1c0f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:26.623 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.621+0000 7fec2293e700 1 -- 172.21.15.184:0/850382469 shutdown_connections 2024-02-19T22:09:26.623 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.621+0000 7fec2293e700 1 -- 172.21.15.184:0/850382469 wait complete. 2024-02-19T22:09:26.623 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.622+0000 7fec2293e700 1 Processor -- start 2024-02-19T22:09:26.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.622+0000 7fec2293e700 1 -- start start 2024-02-19T22:09:26.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.622+0000 7fec2293e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec1c0f5330 0x7fec1c105980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:26.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.622+0000 7fec2293e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c105e70 con 0x7fec1c0f5330 2024-02-19T22:09:26.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.622+0000 7fec1bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec1c0f5330 0x7fec1c105980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:26.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.622+0000 7fec1bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec1c0f5330 0x7fec1c105980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:48802/0 (socket says 172.21.15.184:48802) 2024-02-19T22:09:26.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.622+0000 7fec1bfff700 1 -- 172.21.15.184:0/3553835618 learned_addr learned my addr 172.21.15.184:0/3553835618 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:09:26.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.623+0000 7fec1bfff700 1 -- 172.21.15.184:0/3553835618 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c005040 con 0x7fec1c0f5330 2024-02-19T22:09:26.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.623+0000 7fec1bfff700 1 --2- 172.21.15.184:0/3553835618 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec1c0f5330 0x7fec1c105980 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fec0c005010 tx=0x7fec0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:26.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.623+0000 7fec18ff9700 1 -- 172.21.15.184:0/3553835618 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec0c009320 con 0x7fec1c0f5330 2024-02-19T22:09:26.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.623+0000 7fec2293e700 1 -- 172.21.15.184:0/3553835618 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1c106070 con 0x7fec1c0f5330 2024-02-19T22:09:26.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.623+0000 7fec18ff9700 1 -- 172.21.15.184:0/3553835618 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec0c013380 con 0x7fec1c0f5330 2024-02-19T22:09:26.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.623+0000 7fec18ff9700 1 -- 172.21.15.184:0/3553835618 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec0c014e50 con 0x7fec1c0f5330 2024-02-19T22:09:26.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.623+0000 7fec2293e700 1 -- 172.21.15.184:0/3553835618 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1c106500 con 0x7fec1c0f5330 2024-02-19T22:09:26.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.624+0000 7fec18ff9700 1 -- 172.21.15.184:0/3553835618 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec0c0095f0 con 0x7fec1c0f5330 2024-02-19T22:09:26.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.624+0000 7fec2293e700 1 -- 172.21.15.184:0/3553835618 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1c045ad0 con 0x7fec1c0f5330 2024-02-19T22:09:26.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.624+0000 7fec18ff9700 1 --2- 172.21.15.184:0/3553835618 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fec0802ff10 0x7fec080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:26.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.624+0000 7fec18ff9700 1 -- 172.21.15.184:0/3553835618 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec0c040f80 con 0x7fec1c0f5330 2024-02-19T22:09:26.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.624+0000 7fec1b7fe700 1 --2- 172.21.15.184:0/3553835618 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fec0802ff10 0x7fec080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:26.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.625+0000 7fec1b7fe700 1 --2- 172.21.15.184:0/3553835618 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fec0802ff10 0x7fec080323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fec040096a0 tx=0x7fec04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:26.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.629+0000 7fec18ff9700 1 -- 172.21.15.184:0/3553835618 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec0c0263a0 con 0x7fec1c0f5330 2024-02-19T22:09:26.835 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.834+0000 7fec2293e700 1 -- 172.21.15.184:0/3553835618 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec1c05ea50 con 0x7fec1c0f5330 2024-02-19T22:09:26.836 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.835+0000 7fec18ff9700 1 -- 172.21.15.184:0/3553835618 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fec0c010230 con 0x7fec1c0f5330 2024-02-19T22:09:26.836 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:09:26.836 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:09:26.837 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.836+0000 7fec2293e700 1 -- 172.21.15.184:0/3553835618 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fec0802ff10 msgr2=0x7fec080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:26.837 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.836+0000 7fec2293e700 1 --2- 172.21.15.184:0/3553835618 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fec0802ff10 0x7fec080323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fec040096a0 tx=0x7fec04006b40).stop 2024-02-19T22:09:26.837 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.836+0000 7fec2293e700 1 -- 172.21.15.184:0/3553835618 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec1c0f5330 msgr2=0x7fec1c105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:26.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.836+0000 7fec2293e700 1 --2- 172.21.15.184:0/3553835618 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec1c0f5330 0x7fec1c105980 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fec0c005010 tx=0x7fec0c00b040).stop 2024-02-19T22:09:26.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.836+0000 7fec2293e700 1 -- 172.21.15.184:0/3553835618 shutdown_connections 2024-02-19T22:09:26.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.836+0000 7fec2293e700 1 --2- 172.21.15.184:0/3553835618 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fec0802ff10 0x7fec080323d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:26.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.836+0000 7fec2293e700 1 --2- 172.21.15.184:0/3553835618 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fec1c0f5330 0x7fec1c105980 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:26.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.836+0000 7fec2293e700 1 -- 172.21.15.184:0/3553835618 >> 172.21.15.184:0/3553835618 conn(0x7fec1c0f0a90 msgr2=0x7fec1c0f1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:26.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.837+0000 7fec2293e700 1 -- 172.21.15.184:0/3553835618 shutdown_connections 2024-02-19T22:09:26.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:26.837+0000 7fec2293e700 1 -- 172.21.15.184:0/3553835618 wait complete. 2024-02-19T22:09:26.838 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:09:28.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:25.835604+0000 mgr.smithi176.faoogw (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:28.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:26.836228+0000 mon.smithi176 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.184:0/3553835618' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:09:28.479 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:09:28.479 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:09:30.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:29 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:27.835929+0000 mgr.smithi176.faoogw (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:30.317 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.315+0000 7f748477b700 1 -- 172.21.15.184:0/585484646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f747c0f6510 msgr2=0x7f747c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:30.317 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.315+0000 7f748477b700 1 --2- 172.21.15.184:0/585484646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f747c0f6510 0x7f747c0f6930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f746c009fb0 tx=0x7f746c009b30).stop 2024-02-19T22:09:30.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.315+0000 7f748477b700 1 -- 172.21.15.184:0/585484646 shutdown_connections 2024-02-19T22:09:30.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.315+0000 7f748477b700 1 --2- 172.21.15.184:0/585484646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f747c0f6510 0x7f747c0f6930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:30.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.315+0000 7f748477b700 1 -- 172.21.15.184:0/585484646 >> 172.21.15.184:0/585484646 conn(0x7f747c0f1c70 msgr2=0x7f747c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:30.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.315+0000 7f748477b700 1 -- 172.21.15.184:0/585484646 shutdown_connections 2024-02-19T22:09:30.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.315+0000 7f748477b700 1 -- 172.21.15.184:0/585484646 wait complete. 2024-02-19T22:09:30.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.315+0000 7f748477b700 1 Processor -- start 2024-02-19T22:09:30.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.315+0000 7f748477b700 1 -- start start 2024-02-19T22:09:30.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.316+0000 7f748477b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f747c0f6510 0x7f747c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:30.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.316+0000 7f748477b700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c106cd0 con 0x7f747c0f6510 2024-02-19T22:09:30.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.316+0000 7f7482517700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f747c0f6510 0x7f747c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:30.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.316+0000 7f7482517700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f747c0f6510 0x7f747c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:48828/0 (socket says 172.21.15.184:48828) 2024-02-19T22:09:30.318 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.316+0000 7f7482517700 1 -- 172.21.15.184:0/1676383886 learned_addr learned my addr 172.21.15.184:0/1676383886 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:09:30.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.316+0000 7f7482517700 1 -- 172.21.15.184:0/1676383886 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746c005040 con 0x7f747c0f6510 2024-02-19T22:09:30.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.316+0000 7f7482517700 1 --2- 172.21.15.184:0/1676383886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f747c0f6510 0x7f747c1067e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f746c005010 tx=0x7f746c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:30.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.316+0000 7f7472ffd700 1 -- 172.21.15.184:0/1676383886 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f746c009320 con 0x7f747c0f6510 2024-02-19T22:09:30.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.316+0000 7f7472ffd700 1 -- 172.21.15.184:0/1676383886 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f746c013380 con 0x7f747c0f6510 2024-02-19T22:09:30.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.316+0000 7f7472ffd700 1 -- 172.21.15.184:0/1676383886 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f746c014e50 con 0x7f747c0f6510 2024-02-19T22:09:30.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.316+0000 7f748477b700 1 -- 172.21.15.184:0/1676383886 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747c106ed0 con 0x7f747c0f6510 2024-02-19T22:09:30.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.316+0000 7f748477b700 1 -- 172.21.15.184:0/1676383886 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747c107360 con 0x7f747c0f6510 2024-02-19T22:09:30.320 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.317+0000 7f7472ffd700 1 -- 172.21.15.184:0/1676383886 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f746c0095f0 con 0x7f747c0f6510 2024-02-19T22:09:30.320 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.317+0000 7f748477b700 1 -- 172.21.15.184:0/1676383886 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747c045ad0 con 0x7f747c0f6510 2024-02-19T22:09:30.320 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.317+0000 7f7472ffd700 1 --2- 172.21.15.184:0/1676383886 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f746802ff10 0x7f74680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:30.320 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.317+0000 7f7472ffd700 1 -- 172.21.15.184:0/1676383886 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f746c040f70 con 0x7f747c0f6510 2024-02-19T22:09:30.320 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.318+0000 7f7481d16700 1 --2- 172.21.15.184:0/1676383886 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f746802ff10 0x7f74680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:30.320 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.318+0000 7f7481d16700 1 --2- 172.21.15.184:0/1676383886 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f746802ff10 0x7f74680323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f74780096a0 tx=0x7f7478006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:30.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.322+0000 7f7472ffd700 1 -- 172.21.15.184:0/1676383886 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f746c014420 con 0x7f747c0f6510 2024-02-19T22:09:30.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.526+0000 7f748477b700 1 -- 172.21.15.184:0/1676383886 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f747c108210 con 0x7f747c0f6510 2024-02-19T22:09:30.528 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.527+0000 7f7472ffd700 1 -- 172.21.15.184:0/1676383886 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f746c01d980 con 0x7f747c0f6510 2024-02-19T22:09:30.528 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:09:30.528 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:09:30.530 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.529+0000 7f748477b700 1 -- 172.21.15.184:0/1676383886 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f746802ff10 msgr2=0x7f74680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:30.530 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.529+0000 7f748477b700 1 --2- 172.21.15.184:0/1676383886 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f746802ff10 0x7f74680323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f74780096a0 tx=0x7f7478006b40).stop 2024-02-19T22:09:30.530 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.529+0000 7f748477b700 1 -- 172.21.15.184:0/1676383886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f747c0f6510 msgr2=0x7f747c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:30.530 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.529+0000 7f748477b700 1 --2- 172.21.15.184:0/1676383886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f747c0f6510 0x7f747c1067e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f746c005010 tx=0x7f746c00b040).stop 2024-02-19T22:09:30.530 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.529+0000 7f748477b700 1 -- 172.21.15.184:0/1676383886 shutdown_connections 2024-02-19T22:09:30.530 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.529+0000 7f748477b700 1 --2- 172.21.15.184:0/1676383886 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f746802ff10 0x7f74680323d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:30.530 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.529+0000 7f748477b700 1 --2- 172.21.15.184:0/1676383886 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f747c0f6510 0x7f747c1067e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:30.530 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.529+0000 7f748477b700 1 -- 172.21.15.184:0/1676383886 >> 172.21.15.184:0/1676383886 conn(0x7f747c0f1c70 msgr2=0x7f747c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:30.530 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.529+0000 7f748477b700 1 -- 172.21.15.184:0/1676383886 shutdown_connections 2024-02-19T22:09:30.530 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:30.529+0000 7f748477b700 1 -- 172.21.15.184:0/1676383886 wait complete. 2024-02-19T22:09:30.531 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:09:31.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:30 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:30.528465+0000 mon.smithi176 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.184:0/1676383886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:09:32.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:29.836204+0000 mgr.smithi176.faoogw (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:34.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:31.836592+0000 mgr.smithi176.faoogw (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:36.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:33.836893+0000 mgr.smithi176.faoogw (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:36.664 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:09:36.664 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:09:37.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:35.966750+0000 mon.smithi176 (mon.0) 124 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:09:37.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:35.969754+0000 mon.smithi176 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:09:38.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:35.837178+0000 mgr.smithi176.faoogw (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:40.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:37.837433+0000 mgr.smithi176.faoogw (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:41.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:40 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:39.837768+0000 mgr.smithi176.faoogw (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:43.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.251+0000 7f3d92561700 1 -- 172.21.15.184:0/2467388646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8c0f6510 msgr2=0x7f3d8c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:43.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.251+0000 7f3d92561700 1 --2- 172.21.15.184:0/2467388646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8c0f6510 0x7f3d8c0f6930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f3d7c009fb0 tx=0x7f3d7c009b30).stop 2024-02-19T22:09:43.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.252+0000 7f3d92561700 1 -- 172.21.15.184:0/2467388646 shutdown_connections 2024-02-19T22:09:43.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.252+0000 7f3d92561700 1 --2- 172.21.15.184:0/2467388646 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8c0f6510 0x7f3d8c0f6930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:43.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.252+0000 7f3d92561700 1 -- 172.21.15.184:0/2467388646 >> 172.21.15.184:0/2467388646 conn(0x7f3d8c0f1c70 msgr2=0x7f3d8c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:43.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.252+0000 7f3d92561700 1 -- 172.21.15.184:0/2467388646 shutdown_connections 2024-02-19T22:09:43.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.252+0000 7f3d92561700 1 -- 172.21.15.184:0/2467388646 wait complete. 2024-02-19T22:09:43.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.252+0000 7f3d92561700 1 Processor -- start 2024-02-19T22:09:43.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.252+0000 7f3d92561700 1 -- start start 2024-02-19T22:09:43.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.252+0000 7f3d92561700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8c0f6510 0x7f3d8c106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:43.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.252+0000 7f3d92561700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c106d00 con 0x7f3d8c0f6510 2024-02-19T22:09:43.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.253+0000 7f3d8bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8c0f6510 0x7f3d8c106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:43.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.253+0000 7f3d8bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8c0f6510 0x7f3d8c106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:42462/0 (socket says 172.21.15.184:42462) 2024-02-19T22:09:43.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.253+0000 7f3d8bfff700 1 -- 172.21.15.184:0/621556633 learned_addr learned my addr 172.21.15.184:0/621556633 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:09:43.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.253+0000 7f3d8bfff700 1 -- 172.21.15.184:0/621556633 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c005040 con 0x7f3d8c0f6510 2024-02-19T22:09:43.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.253+0000 7f3d8bfff700 1 --2- 172.21.15.184:0/621556633 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8c0f6510 0x7f3d8c106810 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f3d7c0089f0 tx=0x7f3d7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:43.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.253+0000 7f3d88ff9700 1 -- 172.21.15.184:0/621556633 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d7c009320 con 0x7f3d8c0f6510 2024-02-19T22:09:43.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.253+0000 7f3d88ff9700 1 -- 172.21.15.184:0/621556633 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d7c013380 con 0x7f3d8c0f6510 2024-02-19T22:09:43.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.253+0000 7f3d88ff9700 1 -- 172.21.15.184:0/621556633 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d7c01ed80 con 0x7f3d8c0f6510 2024-02-19T22:09:43.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.253+0000 7f3d92561700 1 -- 172.21.15.184:0/621556633 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8c106f00 con 0x7f3d8c0f6510 2024-02-19T22:09:43.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.254+0000 7f3d92561700 1 -- 172.21.15.184:0/621556633 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8c107390 con 0x7f3d8c0f6510 2024-02-19T22:09:43.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.254+0000 7f3d88ff9700 1 -- 172.21.15.184:0/621556633 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d7c014420 con 0x7f3d8c0f6510 2024-02-19T22:09:43.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.254+0000 7f3d88ff9700 1 --2- 172.21.15.184:0/621556633 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3d7402ffb0 0x7f3d74032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:43.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.254+0000 7f3d88ff9700 1 -- 172.21.15.184:0/621556633 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d7c040e10 con 0x7f3d8c0f6510 2024-02-19T22:09:43.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.255+0000 7f3d92561700 1 -- 172.21.15.184:0/621556633 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8c045ad0 con 0x7f3d8c0f6510 2024-02-19T22:09:43.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.255+0000 7f3d8b7fe700 1 --2- 172.21.15.184:0/621556633 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3d7402ffb0 0x7f3d74032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:43.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.255+0000 7f3d8b7fe700 1 --2- 172.21.15.184:0/621556633 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3d7402ffb0 0x7f3d74032470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f3d800096a0 tx=0x7f3d80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:43.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.259+0000 7f3d88ff9700 1 -- 172.21.15.184:0/621556633 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d7c01c910 con 0x7f3d8c0f6510 2024-02-19T22:09:43.384 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:42 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:41.838077+0000 mgr.smithi176.faoogw (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:43.465 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.464+0000 7f3d92561700 1 -- 172.21.15.184:0/621556633 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d8c1080b0 con 0x7f3d8c0f6510 2024-02-19T22:09:43.465 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.465+0000 7f3d88ff9700 1 -- 172.21.15.184:0/621556633 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3d7c01a230 con 0x7f3d8c0f6510 2024-02-19T22:09:43.466 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:09:43.466 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:09:43.467 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.466+0000 7f3d92561700 1 -- 172.21.15.184:0/621556633 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3d7402ffb0 msgr2=0x7f3d74032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:43.467 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.466+0000 7f3d92561700 1 --2- 172.21.15.184:0/621556633 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3d7402ffb0 0x7f3d74032470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f3d800096a0 tx=0x7f3d80006b40).stop 2024-02-19T22:09:43.467 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.466+0000 7f3d92561700 1 -- 172.21.15.184:0/621556633 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8c0f6510 msgr2=0x7f3d8c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:43.467 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.466+0000 7f3d92561700 1 --2- 172.21.15.184:0/621556633 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8c0f6510 0x7f3d8c106810 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f3d7c0089f0 tx=0x7f3d7c00b040).stop 2024-02-19T22:09:43.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.466+0000 7f3d92561700 1 -- 172.21.15.184:0/621556633 shutdown_connections 2024-02-19T22:09:43.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.466+0000 7f3d92561700 1 --2- 172.21.15.184:0/621556633 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3d7402ffb0 0x7f3d74032470 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:43.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.466+0000 7f3d92561700 1 --2- 172.21.15.184:0/621556633 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d8c0f6510 0x7f3d8c106810 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:43.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.466+0000 7f3d92561700 1 -- 172.21.15.184:0/621556633 >> 172.21.15.184:0/621556633 conn(0x7f3d8c0f1c70 msgr2=0x7f3d8c0f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:43.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.466+0000 7f3d92561700 1 -- 172.21.15.184:0/621556633 shutdown_connections 2024-02-19T22:09:43.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:43.466+0000 7f3d92561700 1 -- 172.21.15.184:0/621556633 wait complete. 2024-02-19T22:09:43.468 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:09:44.384 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:43.466013+0000 mon.smithi176 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.184:0/621556633' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:09:45.091 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:09:45.092 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:09:46.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:43.838427+0000 mgr.smithi176.faoogw (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:46.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.868+0000 7fbfd4828700 1 -- 172.21.15.184:0/3981449912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd00f5410 msgr2=0x7fbfd00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.868+0000 7fbfd4828700 1 --2- 172.21.15.184:0/3981449912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd00f5410 0x7fbfd00f5830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fbfb8009fb0 tx=0x7fbfb8009b30).stop 2024-02-19T22:09:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.868+0000 7fbfd4828700 1 -- 172.21.15.184:0/3981449912 shutdown_connections 2024-02-19T22:09:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.868+0000 7fbfd4828700 1 --2- 172.21.15.184:0/3981449912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd00f5410 0x7fbfd00f5830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.868+0000 7fbfd4828700 1 -- 172.21.15.184:0/3981449912 >> 172.21.15.184:0/3981449912 conn(0x7fbfd00f0b70 msgr2=0x7fbfd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.868+0000 7fbfd4828700 1 -- 172.21.15.184:0/3981449912 shutdown_connections 2024-02-19T22:09:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.869+0000 7fbfd4828700 1 -- 172.21.15.184:0/3981449912 wait complete. 2024-02-19T22:09:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.869+0000 7fbfd4828700 1 Processor -- start 2024-02-19T22:09:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.869+0000 7fbfd4828700 1 -- start start 2024-02-19T22:09:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.869+0000 7fbfd4828700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd00f5410 0x7fbfd0105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:46.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.869+0000 7fbfd4828700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd0105c10 con 0x7fbfd00f5410 2024-02-19T22:09:46.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.870+0000 7fbfce59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd00f5410 0x7fbfd0105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:46.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.870+0000 7fbfce59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd00f5410 0x7fbfd0105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:36236/0 (socket says 172.21.15.184:36236) 2024-02-19T22:09:46.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.870+0000 7fbfce59c700 1 -- 172.21.15.184:0/2632958526 learned_addr learned my addr 172.21.15.184:0/2632958526 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:09:46.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.870+0000 7fbfce59c700 1 -- 172.21.15.184:0/2632958526 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb8005040 con 0x7fbfd00f5410 2024-02-19T22:09:46.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.870+0000 7fbfce59c700 1 --2- 172.21.15.184:0/2632958526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd00f5410 0x7fbfd0105720 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fbfb8005010 tx=0x7fbfb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:46.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.870+0000 7fbfc6ffd700 1 -- 172.21.15.184:0/2632958526 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfb8009320 con 0x7fbfd00f5410 2024-02-19T22:09:46.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.870+0000 7fbfd4828700 1 -- 172.21.15.184:0/2632958526 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd0105e10 con 0x7fbfd00f5410 2024-02-19T22:09:46.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.870+0000 7fbfc6ffd700 1 -- 172.21.15.184:0/2632958526 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbfb8013380 con 0x7fbfd00f5410 2024-02-19T22:09:46.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.870+0000 7fbfc6ffd700 1 -- 172.21.15.184:0/2632958526 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfb8014e50 con 0x7fbfd00f5410 2024-02-19T22:09:46.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.870+0000 7fbfd4828700 1 -- 172.21.15.184:0/2632958526 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd01062a0 con 0x7fbfd00f5410 2024-02-19T22:09:46.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.871+0000 7fbfc6ffd700 1 -- 172.21.15.184:0/2632958526 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbfb80095f0 con 0x7fbfd00f5410 2024-02-19T22:09:46.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.871+0000 7fbfd4828700 1 -- 172.21.15.184:0/2632958526 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd0045ad0 con 0x7fbfd00f5410 2024-02-19T22:09:46.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.871+0000 7fbfc6ffd700 1 --2- 172.21.15.184:0/2632958526 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbfbc02ff10 0x7fbfbc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:46.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.871+0000 7fbfc6ffd700 1 -- 172.21.15.184:0/2632958526 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbfb8040ed0 con 0x7fbfd00f5410 2024-02-19T22:09:46.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.872+0000 7fbfcdd9b700 1 --2- 172.21.15.184:0/2632958526 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbfbc02ff10 0x7fbfbc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:46.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.872+0000 7fbfcdd9b700 1 --2- 172.21.15.184:0/2632958526 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbfbc02ff10 0x7fbfbc0323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fbfc00096a0 tx=0x7fbfc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:46.877 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:46.876+0000 7fbfc6ffd700 1 -- 172.21.15.184:0/2632958526 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbfb8014420 con 0x7fbfd00f5410 2024-02-19T22:09:47.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:47.080+0000 7fbfd4828700 1 -- 172.21.15.184:0/2632958526 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbfd0107150 con 0x7fbfd00f5410 2024-02-19T22:09:47.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:47.081+0000 7fbfc6ffd700 1 -- 172.21.15.184:0/2632958526 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbfb8010230 con 0x7fbfd00f5410 2024-02-19T22:09:47.082 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:09:47.082 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:09:47.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:47.082+0000 7fbfd4828700 1 -- 172.21.15.184:0/2632958526 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbfbc02ff10 msgr2=0x7fbfbc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:47.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:47.082+0000 7fbfd4828700 1 --2- 172.21.15.184:0/2632958526 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbfbc02ff10 0x7fbfbc0323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fbfc00096a0 tx=0x7fbfc0006b40).stop 2024-02-19T22:09:47.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:47.082+0000 7fbfd4828700 1 -- 172.21.15.184:0/2632958526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd00f5410 msgr2=0x7fbfd0105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:47.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:47.082+0000 7fbfd4828700 1 --2- 172.21.15.184:0/2632958526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd00f5410 0x7fbfd0105720 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fbfb8005010 tx=0x7fbfb800b040).stop 2024-02-19T22:09:47.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:47.082+0000 7fbfd4828700 1 -- 172.21.15.184:0/2632958526 shutdown_connections 2024-02-19T22:09:47.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:47.082+0000 7fbfd4828700 1 --2- 172.21.15.184:0/2632958526 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbfbc02ff10 0x7fbfbc0323d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:47.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:47.082+0000 7fbfd4828700 1 --2- 172.21.15.184:0/2632958526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfd00f5410 0x7fbfd0105720 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:47.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:47.082+0000 7fbfd4828700 1 -- 172.21.15.184:0/2632958526 >> 172.21.15.184:0/2632958526 conn(0x7fbfd00f0b70 msgr2=0x7fbfd00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:47.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:47.082+0000 7fbfd4828700 1 -- 172.21.15.184:0/2632958526 shutdown_connections 2024-02-19T22:09:47.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:47.082+0000 7fbfd4828700 1 -- 172.21.15.184:0/2632958526 wait complete. 2024-02-19T22:09:47.084 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:09:48.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:45.838797+0000 mgr.smithi176.faoogw (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:48.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:47.082115+0000 mon.smithi176 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.184:0/2632958526' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:09:48.650 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:09:48.650 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:09:50.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:49 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:47.839141+0000 mgr.smithi176.faoogw (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:50.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.374+0000 7fb28359e700 1 -- 172.21.15.184:0/1220244685 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb284060160 msgr2=0x7fb2840fb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:50.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.374+0000 7fb28359e700 1 --2- 172.21.15.184:0/1220244685 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb284060160 0x7fb2840fb3a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fb274009fb0 tx=0x7fb274009b30).stop 2024-02-19T22:09:50.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.374+0000 7fb28359e700 1 -- 172.21.15.184:0/1220244685 shutdown_connections 2024-02-19T22:09:50.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.374+0000 7fb28359e700 1 --2- 172.21.15.184:0/1220244685 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb284060160 0x7fb2840fb3a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:50.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.374+0000 7fb28359e700 1 -- 172.21.15.184:0/1220244685 >> 172.21.15.184:0/1220244685 conn(0x7fb2840f09a0 msgr2=0x7fb2840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:50.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.374+0000 7fb28359e700 1 -- 172.21.15.184:0/1220244685 shutdown_connections 2024-02-19T22:09:50.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.374+0000 7fb28359e700 1 -- 172.21.15.184:0/1220244685 wait complete. 2024-02-19T22:09:50.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.374+0000 7fb28359e700 1 Processor -- start 2024-02-19T22:09:50.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.374+0000 7fb28359e700 1 -- start start 2024-02-19T22:09:50.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.374+0000 7fb28359e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb284060160 0x7fb2841087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:50.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.374+0000 7fb28359e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb284108cc0 con 0x7fb284060160 2024-02-19T22:09:50.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.375+0000 7fb28259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb284060160 0x7fb2841087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:50.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.375+0000 7fb28259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb284060160 0x7fb2841087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:36256/0 (socket says 172.21.15.184:36256) 2024-02-19T22:09:50.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.375+0000 7fb28259c700 1 -- 172.21.15.184:0/1002145313 learned_addr learned my addr 172.21.15.184:0/1002145313 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:09:50.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.375+0000 7fb28259c700 1 -- 172.21.15.184:0/1002145313 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb274005040 con 0x7fb284060160 2024-02-19T22:09:50.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.375+0000 7fb28259c700 1 --2- 172.21.15.184:0/1002145313 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb284060160 0x7fb2841087d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb274005010 tx=0x7fb27400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:50.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.375+0000 7fb272ffd700 1 -- 172.21.15.184:0/1002145313 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb274009320 con 0x7fb284060160 2024-02-19T22:09:50.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.375+0000 7fb272ffd700 1 -- 172.21.15.184:0/1002145313 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb274013380 con 0x7fb284060160 2024-02-19T22:09:50.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.375+0000 7fb28359e700 1 -- 172.21.15.184:0/1002145313 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb284108ec0 con 0x7fb284060160 2024-02-19T22:09:50.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.375+0000 7fb272ffd700 1 -- 172.21.15.184:0/1002145313 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb274014e50 con 0x7fb284060160 2024-02-19T22:09:50.377 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.375+0000 7fb28359e700 1 -- 172.21.15.184:0/1002145313 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb284109350 con 0x7fb284060160 2024-02-19T22:09:50.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.376+0000 7fb272ffd700 1 -- 172.21.15.184:0/1002145313 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2740263a0 con 0x7fb284060160 2024-02-19T22:09:50.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.376+0000 7fb28359e700 1 -- 172.21.15.184:0/1002145313 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb284045ad0 con 0x7fb284060160 2024-02-19T22:09:50.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.376+0000 7fb272ffd700 1 --2- 172.21.15.184:0/1002145313 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb26c02ff10 0x7fb26c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:50.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.376+0000 7fb272ffd700 1 -- 172.21.15.184:0/1002145313 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb274041140 con 0x7fb284060160 2024-02-19T22:09:50.379 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.376+0000 7fb281d9b700 1 --2- 172.21.15.184:0/1002145313 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb26c02ff10 0x7fb26c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:50.379 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.377+0000 7fb281d9b700 1 --2- 172.21.15.184:0/1002145313 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb26c02ff10 0x7fb26c0323d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fb2780096a0 tx=0x7fb278006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:50.381 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.381+0000 7fb272ffd700 1 -- 172.21.15.184:0/1002145313 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2740126f0 con 0x7fb284060160 2024-02-19T22:09:50.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.588+0000 7fb28359e700 1 -- 172.21.15.184:0/1002145313 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb28410a200 con 0x7fb284060160 2024-02-19T22:09:50.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.589+0000 7fb272ffd700 1 -- 172.21.15.184:0/1002145313 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb274012330 con 0x7fb284060160 2024-02-19T22:09:50.590 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:09:50.590 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:09:50.592 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.590+0000 7fb28359e700 1 -- 172.21.15.184:0/1002145313 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb26c02ff10 msgr2=0x7fb26c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:50.592 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.590+0000 7fb28359e700 1 --2- 172.21.15.184:0/1002145313 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb26c02ff10 0x7fb26c0323d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fb2780096a0 tx=0x7fb278006b40).stop 2024-02-19T22:09:50.592 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.590+0000 7fb28359e700 1 -- 172.21.15.184:0/1002145313 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb284060160 msgr2=0x7fb2841087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:50.592 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.590+0000 7fb28359e700 1 --2- 172.21.15.184:0/1002145313 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb284060160 0x7fb2841087d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb274005010 tx=0x7fb27400b040).stop 2024-02-19T22:09:50.592 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.591+0000 7fb28359e700 1 -- 172.21.15.184:0/1002145313 shutdown_connections 2024-02-19T22:09:50.592 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.591+0000 7fb28359e700 1 --2- 172.21.15.184:0/1002145313 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb26c02ff10 0x7fb26c0323d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:50.592 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.591+0000 7fb28359e700 1 --2- 172.21.15.184:0/1002145313 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb284060160 0x7fb2841087d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:50.592 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.591+0000 7fb28359e700 1 -- 172.21.15.184:0/1002145313 >> 172.21.15.184:0/1002145313 conn(0x7fb2840f09a0 msgr2=0x7fb2840f14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:50.592 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.591+0000 7fb28359e700 1 -- 172.21.15.184:0/1002145313 shutdown_connections 2024-02-19T22:09:50.593 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:50.591+0000 7fb28359e700 1 -- 172.21.15.184:0/1002145313 wait complete. 2024-02-19T22:09:50.593 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:09:51.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:50 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:50.590092+0000 mon.smithi176 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.184:0/1002145313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:09:52.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:51 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:49.839389+0000 mgr.smithi176.faoogw (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:52.252 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:09:52.252 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:09:53.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.997+0000 7feb6d70c700 1 -- 172.21.15.184:0/720203882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb680f5410 msgr2=0x7feb680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:54.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.997+0000 7feb6d70c700 1 --2- 172.21.15.184:0/720203882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb680f5410 0x7feb680f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7feb5c009fb0 tx=0x7feb5c009b30).stop 2024-02-19T22:09:54.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.997+0000 7feb6d70c700 1 -- 172.21.15.184:0/720203882 shutdown_connections 2024-02-19T22:09:54.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.997+0000 7feb6d70c700 1 --2- 172.21.15.184:0/720203882 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb680f5410 0x7feb680f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:54.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.997+0000 7feb6d70c700 1 -- 172.21.15.184:0/720203882 >> 172.21.15.184:0/720203882 conn(0x7feb680f0b70 msgr2=0x7feb680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:54.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.997+0000 7feb6d70c700 1 -- 172.21.15.184:0/720203882 shutdown_connections 2024-02-19T22:09:54.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.997+0000 7feb6d70c700 1 -- 172.21.15.184:0/720203882 wait complete. 2024-02-19T22:09:54.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.998+0000 7feb6d70c700 1 Processor -- start 2024-02-19T22:09:54.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.998+0000 7feb6d70c700 1 -- start start 2024-02-19T22:09:54.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.998+0000 7feb6d70c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb680f5410 0x7feb680fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:54.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.998+0000 7feb6d70c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb680fb960 con 0x7feb680f5410 2024-02-19T22:09:54.001 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.998+0000 7feb66ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb680f5410 0x7feb680fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:54.001 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.998+0000 7feb66ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb680f5410 0x7feb680fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:36268/0 (socket says 172.21.15.184:36268) 2024-02-19T22:09:54.001 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.998+0000 7feb66ffd700 1 -- 172.21.15.184:0/1704056295 learned_addr learned my addr 172.21.15.184:0/1704056295 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:09:54.001 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.998+0000 7feb66ffd700 1 -- 172.21.15.184:0/1704056295 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c005040 con 0x7feb680f5410 2024-02-19T22:09:54.001 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.998+0000 7feb66ffd700 1 --2- 172.21.15.184:0/1704056295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb680f5410 0x7feb680fd4e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7feb5c005010 tx=0x7feb5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:54.001 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.999+0000 7feb4f7fe700 1 -- 172.21.15.184:0/1704056295 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb5c009320 con 0x7feb680f5410 2024-02-19T22:09:54.001 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.999+0000 7feb4f7fe700 1 -- 172.21.15.184:0/1704056295 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb5c013380 con 0x7feb680f5410 2024-02-19T22:09:54.001 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.999+0000 7feb4f7fe700 1 -- 172.21.15.184:0/1704056295 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb5c014e50 con 0x7feb680f5410 2024-02-19T22:09:54.001 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.999+0000 7feb6d70c700 1 -- 172.21.15.184:0/1704056295 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb680fbb00 con 0x7feb680f5410 2024-02-19T22:09:54.001 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.999+0000 7feb6d70c700 1 -- 172.21.15.184:0/1704056295 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb680fbf90 con 0x7feb680f5410 2024-02-19T22:09:54.002 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.999+0000 7feb4f7fe700 1 -- 172.21.15.184:0/1704056295 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb5c0095f0 con 0x7feb680f5410 2024-02-19T22:09:54.002 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:53.999+0000 7feb6d70c700 1 -- 172.21.15.184:0/1704056295 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb68059100 con 0x7feb680f5410 2024-02-19T22:09:54.002 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.000+0000 7feb4f7fe700 1 --2- 172.21.15.184:0/1704056295 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7feb5002ff10 0x7feb500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:54.002 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.000+0000 7feb4f7fe700 1 -- 172.21.15.184:0/1704056295 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb5c040f70 con 0x7feb680f5410 2024-02-19T22:09:54.003 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.000+0000 7feb667fc700 1 --2- 172.21.15.184:0/1704056295 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7feb5002ff10 0x7feb500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:54.003 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.000+0000 7feb667fc700 1 --2- 172.21.15.184:0/1704056295 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7feb5002ff10 0x7feb500323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7feb580096a0 tx=0x7feb58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:54.005 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.004+0000 7feb4f7fe700 1 -- 172.21.15.184:0/1704056295 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb5c014420 con 0x7feb680f5410 2024-02-19T22:09:54.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:51.839626+0000 mgr.smithi176.faoogw (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:54.209 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.209+0000 7feb6d70c700 1 -- 172.21.15.184:0/1704056295 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb680fafa0 con 0x7feb680f5410 2024-02-19T22:09:54.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.209+0000 7feb4f7fe700 1 -- 172.21.15.184:0/1704056295 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feb5c01d980 con 0x7feb680f5410 2024-02-19T22:09:54.210 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:09:54.210 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:09:54.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.211+0000 7feb6d70c700 1 -- 172.21.15.184:0/1704056295 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7feb5002ff10 msgr2=0x7feb500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:54.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.211+0000 7feb6d70c700 1 --2- 172.21.15.184:0/1704056295 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7feb5002ff10 0x7feb500323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7feb580096a0 tx=0x7feb58006b40).stop 2024-02-19T22:09:54.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.211+0000 7feb6d70c700 1 -- 172.21.15.184:0/1704056295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb680f5410 msgr2=0x7feb680fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:54.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.211+0000 7feb6d70c700 1 --2- 172.21.15.184:0/1704056295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb680f5410 0x7feb680fd4e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7feb5c005010 tx=0x7feb5c00b040).stop 2024-02-19T22:09:54.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.211+0000 7feb6d70c700 1 -- 172.21.15.184:0/1704056295 shutdown_connections 2024-02-19T22:09:54.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.211+0000 7feb6d70c700 1 --2- 172.21.15.184:0/1704056295 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7feb5002ff10 0x7feb500323d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:54.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.211+0000 7feb6d70c700 1 --2- 172.21.15.184:0/1704056295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb680f5410 0x7feb680fd4e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:54.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.211+0000 7feb6d70c700 1 -- 172.21.15.184:0/1704056295 >> 172.21.15.184:0/1704056295 conn(0x7feb680f0b70 msgr2=0x7feb6805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:54.213 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.211+0000 7feb6d70c700 1 -- 172.21.15.184:0/1704056295 shutdown_connections 2024-02-19T22:09:54.213 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:54.211+0000 7feb6d70c700 1 -- 172.21.15.184:0/1704056295 wait complete. 2024-02-19T22:09:54.213 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:09:55.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:54 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:54.210604+0000 mon.smithi176 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.184:0/1704056295' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:09:55.898 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:09:55.899 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:09:56.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:55 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:53.839948+0000 mgr.smithi176.faoogw (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:57.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.695+0000 7f4337fff700 1 -- 172.21.15.184:0/3700158352 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43380f5810 msgr2=0x7f43380f5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:57.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.695+0000 7f4337fff700 1 --2- 172.21.15.184:0/3700158352 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43380f5810 0x7f43380f5c30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f4328009fb0 tx=0x7f4328009b30).stop 2024-02-19T22:09:57.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.695+0000 7f4337fff700 1 -- 172.21.15.184:0/3700158352 shutdown_connections 2024-02-19T22:09:57.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.695+0000 7f4337fff700 1 --2- 172.21.15.184:0/3700158352 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43380f5810 0x7f43380f5c30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:57.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.695+0000 7f4337fff700 1 -- 172.21.15.184:0/3700158352 >> 172.21.15.184:0/3700158352 conn(0x7f43380f0120 msgr2=0x7f43380f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:57.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.695+0000 7f4337fff700 1 -- 172.21.15.184:0/3700158352 shutdown_connections 2024-02-19T22:09:57.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.695+0000 7f4337fff700 1 -- 172.21.15.184:0/3700158352 wait complete. 2024-02-19T22:09:57.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.695+0000 7f4337fff700 1 Processor -- start 2024-02-19T22:09:57.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.695+0000 7f4337fff700 1 -- start start 2024-02-19T22:09:57.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.695+0000 7f4337fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43380f5810 0x7f4338105470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:57.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.695+0000 7f4337fff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4338105960 con 0x7f43380f5810 2024-02-19T22:09:57.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.696+0000 7f4336ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43380f5810 0x7f4338105470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:57.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.696+0000 7f4336ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43380f5810 0x7f4338105470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:45012/0 (socket says 172.21.15.184:45012) 2024-02-19T22:09:57.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.696+0000 7f4336ffd700 1 -- 172.21.15.184:0/433729930 learned_addr learned my addr 172.21.15.184:0/433729930 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:09:57.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.696+0000 7f4336ffd700 1 -- 172.21.15.184:0/433729930 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4328005040 con 0x7f43380f5810 2024-02-19T22:09:57.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.696+0000 7f4336ffd700 1 --2- 172.21.15.184:0/433729930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43380f5810 0x7f4338105470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f4328007a10 tx=0x7f4328009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:57.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.696+0000 7f431f7fe700 1 -- 172.21.15.184:0/433729930 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4328018070 con 0x7f43380f5810 2024-02-19T22:09:57.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.696+0000 7f4337fff700 1 -- 172.21.15.184:0/433729930 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4338105b60 con 0x7f43380f5810 2024-02-19T22:09:57.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.696+0000 7f431f7fe700 1 -- 172.21.15.184:0/433729930 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4328013410 con 0x7f43380f5810 2024-02-19T22:09:57.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.696+0000 7f431f7fe700 1 -- 172.21.15.184:0/433729930 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f432801c3c0 con 0x7f43380f5810 2024-02-19T22:09:57.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.696+0000 7f4337fff700 1 -- 172.21.15.184:0/433729930 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4338105ff0 con 0x7f43380f5810 2024-02-19T22:09:57.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.697+0000 7f431f7fe700 1 -- 172.21.15.184:0/433729930 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4328013ca0 con 0x7f43380f5810 2024-02-19T22:09:57.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.697+0000 7f4337fff700 1 -- 172.21.15.184:0/433729930 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4338045ad0 con 0x7f43380f5810 2024-02-19T22:09:57.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.697+0000 7f431f7fe700 1 --2- 172.21.15.184:0/433729930 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f432002ff60 0x7f4320032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:09:57.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.697+0000 7f431f7fe700 1 -- 172.21.15.184:0/433729930 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4328041540 con 0x7f43380f5810 2024-02-19T22:09:57.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.697+0000 7f43367fc700 1 --2- 172.21.15.184:0/433729930 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f432002ff60 0x7f4320032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:09:57.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.698+0000 7f43367fc700 1 --2- 172.21.15.184:0/433729930 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f432002ff60 0x7f4320032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f432c0096a0 tx=0x7f432c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:09:57.703 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.702+0000 7f431f7fe700 1 -- 172.21.15.184:0/433729930 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4328025d20 con 0x7f43380f5810 2024-02-19T22:09:57.907 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.906+0000 7f4337fff700 1 -- 172.21.15.184:0/433729930 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4338106ea0 con 0x7f43380f5810 2024-02-19T22:09:57.908 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.907+0000 7f431f7fe700 1 -- 172.21.15.184:0/433729930 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4328027e00 con 0x7f43380f5810 2024-02-19T22:09:57.908 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:09:57.908 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:09:57.909 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.908+0000 7f4337fff700 1 -- 172.21.15.184:0/433729930 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f432002ff60 msgr2=0x7f4320032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:57.909 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.908+0000 7f4337fff700 1 --2- 172.21.15.184:0/433729930 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f432002ff60 0x7f4320032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f432c0096a0 tx=0x7f432c006b40).stop 2024-02-19T22:09:57.909 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.908+0000 7f4337fff700 1 -- 172.21.15.184:0/433729930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43380f5810 msgr2=0x7f4338105470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:09:57.909 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.908+0000 7f4337fff700 1 --2- 172.21.15.184:0/433729930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43380f5810 0x7f4338105470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f4328007a10 tx=0x7f4328009b30).stop 2024-02-19T22:09:57.909 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.908+0000 7f4337fff700 1 -- 172.21.15.184:0/433729930 shutdown_connections 2024-02-19T22:09:57.909 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.908+0000 7f4337fff700 1 --2- 172.21.15.184:0/433729930 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f432002ff60 0x7f4320032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:57.910 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.908+0000 7f4337fff700 1 --2- 172.21.15.184:0/433729930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43380f5810 0x7f4338105470 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:09:57.910 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.908+0000 7f4337fff700 1 -- 172.21.15.184:0/433729930 >> 172.21.15.184:0/433729930 conn(0x7f43380f0120 msgr2=0x7f43380f0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:09:57.910 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.908+0000 7f4337fff700 1 -- 172.21.15.184:0/433729930 shutdown_connections 2024-02-19T22:09:57.910 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:09:57.908+0000 7f4337fff700 1 -- 172.21.15.184:0/433729930 wait complete. 2024-02-19T22:09:57.910 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:09:58.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:55.840327+0000 mgr.smithi176.faoogw (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:09:59.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:58 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:09:57.907999+0000 mon.smithi176 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.184:0/433729930' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:09:59.531 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:09:59.532 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:00.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:09:59 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:57.840619+0000 mgr.smithi176.faoogw (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:01.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.255+0000 7f729a5fa700 1 -- 172.21.15.184:0/1270422052 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72940f62c0 msgr2=0x7f72940f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:01.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.255+0000 7f729a5fa700 1 --2- 172.21.15.184:0/1270422052 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72940f62c0 0x7f72940f66e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f7290009fb0 tx=0x7f7290009b30).stop 2024-02-19T22:10:01.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.255+0000 7f729a5fa700 1 -- 172.21.15.184:0/1270422052 shutdown_connections 2024-02-19T22:10:01.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.255+0000 7f729a5fa700 1 --2- 172.21.15.184:0/1270422052 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72940f62c0 0x7f72940f66e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:01.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.255+0000 7f729a5fa700 1 -- 172.21.15.184:0/1270422052 >> 172.21.15.184:0/1270422052 conn(0x7f72940f1ac0 msgr2=0x7f72940f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:01.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.255+0000 7f729a5fa700 1 -- 172.21.15.184:0/1270422052 shutdown_connections 2024-02-19T22:10:01.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.255+0000 7f729a5fa700 1 -- 172.21.15.184:0/1270422052 wait complete. 2024-02-19T22:10:01.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.255+0000 7f729a5fa700 1 Processor -- start 2024-02-19T22:10:01.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.255+0000 7f729a5fa700 1 -- start start 2024-02-19T22:10:01.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.255+0000 7f729a5fa700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72940f62c0 0x7f72941065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:01.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.255+0000 7f729a5fa700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7294106ad0 con 0x7f72940f62c0 2024-02-19T22:10:01.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.256+0000 7f72995f8700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72940f62c0 0x7f72941065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:01.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.256+0000 7f72995f8700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72940f62c0 0x7f72941065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:45022/0 (socket says 172.21.15.184:45022) 2024-02-19T22:10:01.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.256+0000 7f72995f8700 1 -- 172.21.15.184:0/219614722 learned_addr learned my addr 172.21.15.184:0/219614722 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:01.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.256+0000 7f72995f8700 1 -- 172.21.15.184:0/219614722 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7290005040 con 0x7f72940f62c0 2024-02-19T22:10:01.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.256+0000 7f72995f8700 1 --2- 172.21.15.184:0/219614722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72940f62c0 0x7f72941065e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f72900089f0 tx=0x7f729000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:01.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.256+0000 7f7289ffb700 1 -- 172.21.15.184:0/219614722 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7290009320 con 0x7f72940f62c0 2024-02-19T22:10:01.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.256+0000 7f729a5fa700 1 -- 172.21.15.184:0/219614722 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7294106cd0 con 0x7f72940f62c0 2024-02-19T22:10:01.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.256+0000 7f7289ffb700 1 -- 172.21.15.184:0/219614722 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7290013380 con 0x7f72940f62c0 2024-02-19T22:10:01.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.256+0000 7f7289ffb700 1 -- 172.21.15.184:0/219614722 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7290014e20 con 0x7f72940f62c0 2024-02-19T22:10:01.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.256+0000 7f729a5fa700 1 -- 172.21.15.184:0/219614722 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7294107160 con 0x7f72940f62c0 2024-02-19T22:10:01.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.257+0000 7f7289ffb700 1 -- 172.21.15.184:0/219614722 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72900095f0 con 0x7f72940f62c0 2024-02-19T22:10:01.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.257+0000 7f729a5fa700 1 -- 172.21.15.184:0/219614722 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7294045ad0 con 0x7f72940f62c0 2024-02-19T22:10:01.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.257+0000 7f7289ffb700 1 --2- 172.21.15.184:0/219614722 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f728002ff10 0x7f72800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:01.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.257+0000 7f7289ffb700 1 -- 172.21.15.184:0/219614722 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7290040f50 con 0x7f72940f62c0 2024-02-19T22:10:01.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.257+0000 7f7298df7700 1 --2- 172.21.15.184:0/219614722 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f728002ff10 0x7f72800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:01.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.258+0000 7f7298df7700 1 --2- 172.21.15.184:0/219614722 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f728002ff10 0x7f72800323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f72840096a0 tx=0x7f7284006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:01.263 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.262+0000 7f7289ffb700 1 -- 172.21.15.184:0/219614722 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7290014420 con 0x7f72940f62c0 2024-02-19T22:10:01.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.469+0000 7f729a5fa700 1 -- 172.21.15.184:0/219614722 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7294108010 con 0x7f72940f62c0 2024-02-19T22:10:01.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.470+0000 7f7289ffb700 1 -- 172.21.15.184:0/219614722 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f729001d980 con 0x7f72940f62c0 2024-02-19T22:10:01.471 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:01.471 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:01.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.472+0000 7f729a5fa700 1 -- 172.21.15.184:0/219614722 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f728002ff10 msgr2=0x7f72800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:01.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.472+0000 7f729a5fa700 1 --2- 172.21.15.184:0/219614722 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f728002ff10 0x7f72800323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f72840096a0 tx=0x7f7284006b40).stop 2024-02-19T22:10:01.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.472+0000 7f729a5fa700 1 -- 172.21.15.184:0/219614722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72940f62c0 msgr2=0x7f72941065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:01.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.472+0000 7f729a5fa700 1 --2- 172.21.15.184:0/219614722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72940f62c0 0x7f72941065e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f72900089f0 tx=0x7f729000b040).stop 2024-02-19T22:10:01.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.472+0000 7f729a5fa700 1 -- 172.21.15.184:0/219614722 shutdown_connections 2024-02-19T22:10:01.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.472+0000 7f729a5fa700 1 --2- 172.21.15.184:0/219614722 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f728002ff10 0x7f72800323d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:01.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.472+0000 7f729a5fa700 1 --2- 172.21.15.184:0/219614722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f72940f62c0 0x7f72941065e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:01.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.472+0000 7f729a5fa700 1 -- 172.21.15.184:0/219614722 >> 172.21.15.184:0/219614722 conn(0x7f72940f1ac0 msgr2=0x7f72940f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:01.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.472+0000 7f729a5fa700 1 -- 172.21.15.184:0/219614722 shutdown_connections 2024-02-19T22:10:01.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:01.472+0000 7f729a5fa700 1 -- 172.21.15.184:0/219614722 wait complete. 2024-02-19T22:10:01.474 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:02.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:09:59.840944+0000 mgr.smithi176.faoogw (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:02.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:01.471425+0000 mon.smithi176 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.184:0/219614722' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:03.059 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:03.060 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:04.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:01.841306+0000 mgr.smithi176.faoogw (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:04.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.762+0000 7f669789f700 1 -- 172.21.15.184:0/952894779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66900f6510 msgr2=0x7f66900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:04.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.762+0000 7f669789f700 1 --2- 172.21.15.184:0/952894779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66900f6510 0x7f66900f6930 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f6680009fb0 tx=0x7f6680009b30).stop 2024-02-19T22:10:04.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.762+0000 7f669789f700 1 -- 172.21.15.184:0/952894779 shutdown_connections 2024-02-19T22:10:04.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.762+0000 7f669789f700 1 --2- 172.21.15.184:0/952894779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66900f6510 0x7f66900f6930 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:04.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.762+0000 7f669789f700 1 -- 172.21.15.184:0/952894779 >> 172.21.15.184:0/952894779 conn(0x7f66900f1c70 msgr2=0x7f66900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:04.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.762+0000 7f669789f700 1 -- 172.21.15.184:0/952894779 shutdown_connections 2024-02-19T22:10:04.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.762+0000 7f669789f700 1 -- 172.21.15.184:0/952894779 wait complete. 2024-02-19T22:10:04.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.762+0000 7f669789f700 1 Processor -- start 2024-02-19T22:10:04.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.762+0000 7f669789f700 1 -- start start 2024-02-19T22:10:04.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.762+0000 7f669789f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66900f6510 0x7f669006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:04.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.762+0000 7f669789f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6690069e20 con 0x7f66900f6510 2024-02-19T22:10:04.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.763+0000 7f669563b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66900f6510 0x7f669006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:04.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.763+0000 7f669563b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66900f6510 0x7f669006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:45044/0 (socket says 172.21.15.184:45044) 2024-02-19T22:10:04.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.763+0000 7f669563b700 1 -- 172.21.15.184:0/3755207289 learned_addr learned my addr 172.21.15.184:0/3755207289 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:04.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.763+0000 7f669563b700 1 -- 172.21.15.184:0/3755207289 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6680005040 con 0x7f66900f6510 2024-02-19T22:10:04.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.763+0000 7f669563b700 1 --2- 172.21.15.184:0/3755207289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66900f6510 0x7f669006b9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f66800089f0 tx=0x7f668000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:04.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.763+0000 7f6685ffb700 1 -- 172.21.15.184:0/3755207289 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6680009320 con 0x7f66900f6510 2024-02-19T22:10:04.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.763+0000 7f669789f700 1 -- 172.21.15.184:0/3755207289 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6690069fc0 con 0x7f66900f6510 2024-02-19T22:10:04.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.763+0000 7f6685ffb700 1 -- 172.21.15.184:0/3755207289 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6680013380 con 0x7f66900f6510 2024-02-19T22:10:04.765 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.763+0000 7f6685ffb700 1 -- 172.21.15.184:0/3755207289 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f668001ed80 con 0x7f66900f6510 2024-02-19T22:10:04.766 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.763+0000 7f669789f700 1 -- 172.21.15.184:0/3755207289 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669006a450 con 0x7f66900f6510 2024-02-19T22:10:04.766 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.764+0000 7f6685ffb700 1 -- 172.21.15.184:0/3755207289 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f668001e420 con 0x7f66900f6510 2024-02-19T22:10:04.766 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.764+0000 7f6685ffb700 1 --2- 172.21.15.184:0/3755207289 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f667c02ff60 0x7f667c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:04.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.764+0000 7f669789f700 1 -- 172.21.15.184:0/3755207289 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6690059100 con 0x7f66900f6510 2024-02-19T22:10:04.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.764+0000 7f6685ffb700 1 -- 172.21.15.184:0/3755207289 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6680040ed0 con 0x7f66900f6510 2024-02-19T22:10:04.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.764+0000 7f6694e3a700 1 --2- 172.21.15.184:0/3755207289 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f667c02ff60 0x7f667c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:04.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.765+0000 7f6694e3a700 1 --2- 172.21.15.184:0/3755207289 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f667c02ff60 0x7f667c032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f668c0096a0 tx=0x7f668c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:04.770 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.769+0000 7f6685ffb700 1 -- 172.21.15.184:0/3755207289 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66800095f0 con 0x7f66900f6510 2024-02-19T22:10:04.973 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.972+0000 7f669789f700 1 -- 172.21.15.184:0/3755207289 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66900fc0a0 con 0x7f66900f6510 2024-02-19T22:10:04.974 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.973+0000 7f6685ffb700 1 -- 172.21.15.184:0/3755207289 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f66800095f0 con 0x7f66900f6510 2024-02-19T22:10:04.974 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:04.974 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:04.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.974+0000 7f669789f700 1 -- 172.21.15.184:0/3755207289 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f667c02ff60 msgr2=0x7f667c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:04.976 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.974+0000 7f669789f700 1 --2- 172.21.15.184:0/3755207289 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f667c02ff60 0x7f667c032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f668c0096a0 tx=0x7f668c006b40).stop 2024-02-19T22:10:04.976 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.974+0000 7f669789f700 1 -- 172.21.15.184:0/3755207289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66900f6510 msgr2=0x7f669006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:04.976 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.974+0000 7f669789f700 1 --2- 172.21.15.184:0/3755207289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66900f6510 0x7f669006b9a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f66800089f0 tx=0x7f668000b040).stop 2024-02-19T22:10:04.976 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.975+0000 7f669789f700 1 -- 172.21.15.184:0/3755207289 shutdown_connections 2024-02-19T22:10:04.976 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.975+0000 7f669789f700 1 --2- 172.21.15.184:0/3755207289 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f667c02ff60 0x7f667c032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:04.976 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.975+0000 7f669789f700 1 --2- 172.21.15.184:0/3755207289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66900f6510 0x7f669006b9a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:04.976 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.975+0000 7f669789f700 1 -- 172.21.15.184:0/3755207289 >> 172.21.15.184:0/3755207289 conn(0x7f66900f1c70 msgr2=0x7f66900f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:04.976 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.975+0000 7f669789f700 1 -- 172.21.15.184:0/3755207289 shutdown_connections 2024-02-19T22:10:04.976 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:04.975+0000 7f669789f700 1 -- 172.21.15.184:0/3755207289 wait complete. 2024-02-19T22:10:04.977 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:06.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:03.841566+0000 mgr.smithi176.faoogw (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:06.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:04.974227+0000 mon.smithi176 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.184:0/3755207289' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:06.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:06.471 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:08.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:05.841796+0000 mgr.smithi176.faoogw (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.600+0000 7fb6b4874700 1 -- 172.21.15.184:0/1499302383 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6b0060220 msgr2=0x7fb6b00f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.600+0000 7fb6b4874700 1 --2- 172.21.15.184:0/1499302383 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6b0060220 0x7fb6b00f9150 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fb6a0009fb0 tx=0x7fb6a0009b30).stop 2024-02-19T22:10:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.600+0000 7fb6b4874700 1 -- 172.21.15.184:0/1499302383 shutdown_connections 2024-02-19T22:10:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.600+0000 7fb6b4874700 1 --2- 172.21.15.184:0/1499302383 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6b0060220 0x7fb6b00f9150 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.600+0000 7fb6b4874700 1 -- 172.21.15.184:0/1499302383 >> 172.21.15.184:0/1499302383 conn(0x7fb6b00f0b70 msgr2=0x7fb6b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.600+0000 7fb6b4874700 1 -- 172.21.15.184:0/1499302383 shutdown_connections 2024-02-19T22:10:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.600+0000 7fb6b4874700 1 -- 172.21.15.184:0/1499302383 wait complete. 2024-02-19T22:10:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.601+0000 7fb6b4874700 1 Processor -- start 2024-02-19T22:10:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.601+0000 7fb6b4874700 1 -- start start 2024-02-19T22:10:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.601+0000 7fb6b4874700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6b0060220 0x7fb6b00f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:08.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.601+0000 7fb6b4874700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b00f5050 con 0x7fb6b0060220 2024-02-19T22:10:08.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.601+0000 7fb6ae59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6b0060220 0x7fb6b00f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:08.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.601+0000 7fb6ae59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6b0060220 0x7fb6b00f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:40796/0 (socket says 172.21.15.184:40796) 2024-02-19T22:10:08.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.601+0000 7fb6ae59c700 1 -- 172.21.15.184:0/381502077 learned_addr learned my addr 172.21.15.184:0/381502077 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:08.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.601+0000 7fb6ae59c700 1 -- 172.21.15.184:0/381502077 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a0005040 con 0x7fb6b0060220 2024-02-19T22:10:08.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.601+0000 7fb6ae59c700 1 --2- 172.21.15.184:0/381502077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6b0060220 0x7fb6b00f6bd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb6a00089f0 tx=0x7fb6a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:08.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.601+0000 7fb69effd700 1 -- 172.21.15.184:0/381502077 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6a0009320 con 0x7fb6b0060220 2024-02-19T22:10:08.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.601+0000 7fb6b4874700 1 -- 172.21.15.184:0/381502077 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b00f51f0 con 0x7fb6b0060220 2024-02-19T22:10:08.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.602+0000 7fb69effd700 1 -- 172.21.15.184:0/381502077 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6a0013380 con 0x7fb6b0060220 2024-02-19T22:10:08.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.602+0000 7fb69effd700 1 -- 172.21.15.184:0/381502077 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6a0014e20 con 0x7fb6b0060220 2024-02-19T22:10:08.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.602+0000 7fb6b4874700 1 -- 172.21.15.184:0/381502077 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b00f5680 con 0x7fb6b0060220 2024-02-19T22:10:08.605 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.602+0000 7fb69effd700 1 -- 172.21.15.184:0/381502077 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6a0014420 con 0x7fb6b0060220 2024-02-19T22:10:08.605 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.602+0000 7fb6b4874700 1 -- 172.21.15.184:0/381502077 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b0059100 con 0x7fb6b0060220 2024-02-19T22:10:08.605 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.603+0000 7fb69effd700 1 --2- 172.21.15.184:0/381502077 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb69802ff10 0x7fb6980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:08.605 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.603+0000 7fb69effd700 1 -- 172.21.15.184:0/381502077 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb6a0040e70 con 0x7fb6b0060220 2024-02-19T22:10:08.606 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.603+0000 7fb6add9b700 1 --2- 172.21.15.184:0/381502077 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb69802ff10 0x7fb6980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:08.606 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.603+0000 7fb6add9b700 1 --2- 172.21.15.184:0/381502077 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb69802ff10 0x7fb6980323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fb6a40096a0 tx=0x7fb6a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:08.608 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.607+0000 7fb69effd700 1 -- 172.21.15.184:0/381502077 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6a001d980 con 0x7fb6b0060220 2024-02-19T22:10:08.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.817+0000 7fb6b4874700 1 -- 172.21.15.184:0/381502077 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6b00fc090 con 0x7fb6b0060220 2024-02-19T22:10:08.819 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.818+0000 7fb69effd700 1 -- 172.21.15.184:0/381502077 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb6a001d380 con 0x7fb6b0060220 2024-02-19T22:10:08.819 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:08.819 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:08.820 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.819+0000 7fb6b4874700 1 -- 172.21.15.184:0/381502077 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb69802ff10 msgr2=0x7fb6980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:08.820 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.819+0000 7fb6b4874700 1 --2- 172.21.15.184:0/381502077 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb69802ff10 0x7fb6980323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fb6a40096a0 tx=0x7fb6a4006b40).stop 2024-02-19T22:10:08.820 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.819+0000 7fb6b4874700 1 -- 172.21.15.184:0/381502077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6b0060220 msgr2=0x7fb6b00f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:08.820 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.819+0000 7fb6b4874700 1 --2- 172.21.15.184:0/381502077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6b0060220 0x7fb6b00f6bd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb6a00089f0 tx=0x7fb6a000b040).stop 2024-02-19T22:10:08.821 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.819+0000 7fb6b4874700 1 -- 172.21.15.184:0/381502077 shutdown_connections 2024-02-19T22:10:08.821 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.819+0000 7fb6b4874700 1 --2- 172.21.15.184:0/381502077 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb69802ff10 0x7fb6980323d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:08.821 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.819+0000 7fb6b4874700 1 --2- 172.21.15.184:0/381502077 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6b0060220 0x7fb6b00f6bd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:08.821 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.819+0000 7fb6b4874700 1 -- 172.21.15.184:0/381502077 >> 172.21.15.184:0/381502077 conn(0x7fb6b00f0b70 msgr2=0x7fb6b00f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:08.821 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.819+0000 7fb6b4874700 1 -- 172.21.15.184:0/381502077 shutdown_connections 2024-02-19T22:10:08.821 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:08.819+0000 7fb6b4874700 1 -- 172.21.15.184:0/381502077 wait complete. 2024-02-19T22:10:08.821 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:10.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:07.842093+0000 mgr.smithi176.faoogw (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:10.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:08.819138+0000 mon.smithi176 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.184:0/381502077' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:10.406 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:10.406 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:12.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:09.842432+0000 mgr.smithi176.faoogw (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:12.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.282+0000 7f87b7b78700 1 -- 172.21.15.184:0/2880185515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87b00f5410 msgr2=0x7f87b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:12.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.282+0000 7f87b7b78700 1 --2- 172.21.15.184:0/2880185515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87b00f5410 0x7f87b00f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f87a0009fb0 tx=0x7f87a0009b30).stop 2024-02-19T22:10:12.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.282+0000 7f87b7b78700 1 -- 172.21.15.184:0/2880185515 shutdown_connections 2024-02-19T22:10:12.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.282+0000 7f87b7b78700 1 --2- 172.21.15.184:0/2880185515 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87b00f5410 0x7f87b00f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:12.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.282+0000 7f87b7b78700 1 -- 172.21.15.184:0/2880185515 >> 172.21.15.184:0/2880185515 conn(0x7f87b00f0b70 msgr2=0x7f87b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:12.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.282+0000 7f87b7b78700 1 -- 172.21.15.184:0/2880185515 shutdown_connections 2024-02-19T22:10:12.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.282+0000 7f87b7b78700 1 -- 172.21.15.184:0/2880185515 wait complete. 2024-02-19T22:10:12.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.283+0000 7f87b7b78700 1 Processor -- start 2024-02-19T22:10:12.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.283+0000 7f87b7b78700 1 -- start start 2024-02-19T22:10:12.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.283+0000 7f87b7b78700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87b00f5410 0x7f87b01094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:12.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.283+0000 7f87b7b78700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b0060280 con 0x7f87b00f5410 2024-02-19T22:10:12.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.283+0000 7f87b5914700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87b00f5410 0x7f87b01094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:12.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.283+0000 7f87b5914700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87b00f5410 0x7f87b01094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:40814/0 (socket says 172.21.15.184:40814) 2024-02-19T22:10:12.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.283+0000 7f87b5914700 1 -- 172.21.15.184:0/1570635542 learned_addr learned my addr 172.21.15.184:0/1570635542 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:12.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.283+0000 7f87b5914700 1 -- 172.21.15.184:0/1570635542 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a0005040 con 0x7f87b00f5410 2024-02-19T22:10:12.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.283+0000 7f87b5914700 1 --2- 172.21.15.184:0/1570635542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87b00f5410 0x7f87b01094a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f87a0005010 tx=0x7f87a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:12.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.284+0000 7f87a67fc700 1 -- 172.21.15.184:0/1570635542 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87a0009320 con 0x7f87b00f5410 2024-02-19T22:10:12.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.284+0000 7f87b7b78700 1 -- 172.21.15.184:0/1570635542 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b0060480 con 0x7f87b00f5410 2024-02-19T22:10:12.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.284+0000 7f87b7b78700 1 -- 172.21.15.184:0/1570635542 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b0109bd0 con 0x7f87b00f5410 2024-02-19T22:10:12.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.284+0000 7f87a67fc700 1 -- 172.21.15.184:0/1570635542 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87a0013380 con 0x7f87b00f5410 2024-02-19T22:10:12.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.284+0000 7f87a67fc700 1 -- 172.21.15.184:0/1570635542 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87a0014df0 con 0x7f87b00f5410 2024-02-19T22:10:12.287 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.285+0000 7f87a67fc700 1 -- 172.21.15.184:0/1570635542 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87a00095f0 con 0x7f87b00f5410 2024-02-19T22:10:12.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.285+0000 7f87b7b78700 1 -- 172.21.15.184:0/1570635542 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b0045ad0 con 0x7f87b00f5410 2024-02-19T22:10:12.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.285+0000 7f87a67fc700 1 --2- 172.21.15.184:0/1570635542 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f879c02ff10 0x7f879c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:12.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.285+0000 7f87a67fc700 1 -- 172.21.15.184:0/1570635542 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87a0040e90 con 0x7f87b00f5410 2024-02-19T22:10:12.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.285+0000 7f87b5113700 1 --2- 172.21.15.184:0/1570635542 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f879c02ff10 0x7f879c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:12.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.285+0000 7f87b5113700 1 --2- 172.21.15.184:0/1570635542 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f879c02ff10 0x7f879c0323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f87ac0096a0 tx=0x7f87ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:12.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.289+0000 7f87a67fc700 1 -- 172.21.15.184:0/1570635542 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87a0014420 con 0x7f87b00f5410 2024-02-19T22:10:12.496 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.495+0000 7f87b7b78700 1 -- 172.21.15.184:0/1570635542 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87b00fafa0 con 0x7f87b00f5410 2024-02-19T22:10:12.497 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.496+0000 7f87a67fc700 1 -- 172.21.15.184:0/1570635542 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f87a001d9d0 con 0x7f87b00f5410 2024-02-19T22:10:12.497 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:12.498 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:12.498 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.498+0000 7f87b7b78700 1 -- 172.21.15.184:0/1570635542 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f879c02ff10 msgr2=0x7f879c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:12.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.498+0000 7f87b7b78700 1 --2- 172.21.15.184:0/1570635542 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f879c02ff10 0x7f879c0323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f87ac0096a0 tx=0x7f87ac006b40).stop 2024-02-19T22:10:12.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.498+0000 7f87b7b78700 1 -- 172.21.15.184:0/1570635542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87b00f5410 msgr2=0x7f87b01094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:12.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.498+0000 7f87b7b78700 1 --2- 172.21.15.184:0/1570635542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87b00f5410 0x7f87b01094a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f87a0005010 tx=0x7f87a000b040).stop 2024-02-19T22:10:12.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.498+0000 7f87b7b78700 1 -- 172.21.15.184:0/1570635542 shutdown_connections 2024-02-19T22:10:12.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.498+0000 7f87b7b78700 1 --2- 172.21.15.184:0/1570635542 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f879c02ff10 0x7f879c0323d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:12.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.498+0000 7f87b7b78700 1 --2- 172.21.15.184:0/1570635542 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87b00f5410 0x7f87b01094a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:12.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.498+0000 7f87b7b78700 1 -- 172.21.15.184:0/1570635542 >> 172.21.15.184:0/1570635542 conn(0x7f87b00f0b70 msgr2=0x7f87b00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:12.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.498+0000 7f87b7b78700 1 -- 172.21.15.184:0/1570635542 shutdown_connections 2024-02-19T22:10:12.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:12.498+0000 7f87b7b78700 1 -- 172.21.15.184:0/1570635542 wait complete. 2024-02-19T22:10:12.500 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:13.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:12 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:12.497318+0000 mon.smithi176 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.184:0/1570635542' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:14.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:13 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:11.842826+0000 mgr.smithi176.faoogw (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:14.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:14.314 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:16.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:13.843198+0000 mgr.smithi176.faoogw (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:16.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.144+0000 7f27c4348700 1 -- 172.21.15.184:0/1688974869 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27bc0f5410 msgr2=0x7f27bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:16.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.144+0000 7f27c4348700 1 --2- 172.21.15.184:0/1688974869 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27bc0f5410 0x7f27bc0f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f27b8009fb0 tx=0x7f27b8009b30).stop 2024-02-19T22:10:16.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.144+0000 7f27c4348700 1 -- 172.21.15.184:0/1688974869 shutdown_connections 2024-02-19T22:10:16.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.144+0000 7f27c4348700 1 --2- 172.21.15.184:0/1688974869 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27bc0f5410 0x7f27bc0f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:16.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.144+0000 7f27c4348700 1 -- 172.21.15.184:0/1688974869 >> 172.21.15.184:0/1688974869 conn(0x7f27bc0f0b70 msgr2=0x7f27bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:16.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.144+0000 7f27c4348700 1 -- 172.21.15.184:0/1688974869 shutdown_connections 2024-02-19T22:10:16.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.144+0000 7f27c4348700 1 -- 172.21.15.184:0/1688974869 wait complete. 2024-02-19T22:10:16.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.145+0000 7f27c4348700 1 Processor -- start 2024-02-19T22:10:16.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.145+0000 7f27c4348700 1 -- start start 2024-02-19T22:10:16.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.145+0000 7f27c4348700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27bc0f5410 0x7f27bc109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:16.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.145+0000 7f27c4348700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc060220 con 0x7f27bc0f5410 2024-02-19T22:10:16.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.145+0000 7f27c20e4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27bc0f5410 0x7f27bc109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:16.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.145+0000 7f27c20e4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27bc0f5410 0x7f27bc109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:40826/0 (socket says 172.21.15.184:40826) 2024-02-19T22:10:16.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.145+0000 7f27c20e4700 1 -- 172.21.15.184:0/1055669738 learned_addr learned my addr 172.21.15.184:0/1055669738 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:16.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.146+0000 7f27c20e4700 1 -- 172.21.15.184:0/1055669738 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b8005040 con 0x7f27bc0f5410 2024-02-19T22:10:16.148 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.146+0000 7f27c20e4700 1 --2- 172.21.15.184:0/1055669738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27bc0f5410 0x7f27bc109500 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f27b8005010 tx=0x7f27b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:16.148 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.146+0000 7f27b2ffd700 1 -- 172.21.15.184:0/1055669738 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27b8009320 con 0x7f27bc0f5410 2024-02-19T22:10:16.148 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.146+0000 7f27c4348700 1 -- 172.21.15.184:0/1055669738 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc060420 con 0x7f27bc0f5410 2024-02-19T22:10:16.148 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.146+0000 7f27b2ffd700 1 -- 172.21.15.184:0/1055669738 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27b8013380 con 0x7f27bc0f5410 2024-02-19T22:10:16.148 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.146+0000 7f27b2ffd700 1 -- 172.21.15.184:0/1055669738 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27b8014e50 con 0x7f27bc0f5410 2024-02-19T22:10:16.148 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.146+0000 7f27c4348700 1 -- 172.21.15.184:0/1055669738 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc109bb0 con 0x7f27bc0f5410 2024-02-19T22:10:16.149 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.147+0000 7f27b2ffd700 1 -- 172.21.15.184:0/1055669738 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27b80263a0 con 0x7f27bc0f5410 2024-02-19T22:10:16.149 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.147+0000 7f27b2ffd700 1 --2- 172.21.15.184:0/1055669738 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f27a802ff10 0x7f27a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:16.149 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.147+0000 7f27c4348700 1 -- 172.21.15.184:0/1055669738 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27bc059100 con 0x7f27bc0f5410 2024-02-19T22:10:16.149 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.147+0000 7f27b2ffd700 1 -- 172.21.15.184:0/1055669738 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27b8040fd0 con 0x7f27bc0f5410 2024-02-19T22:10:16.149 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.147+0000 7f27c18e3700 1 --2- 172.21.15.184:0/1055669738 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f27a802ff10 0x7f27a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:16.149 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.148+0000 7f27c18e3700 1 --2- 172.21.15.184:0/1055669738 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f27a802ff10 0x7f27a80323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f27ac0096a0 tx=0x7f27ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:16.152 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.152+0000 7f27b2ffd700 1 -- 172.21.15.184:0/1055669738 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27b8012920 con 0x7f27bc0f5410 2024-02-19T22:10:16.358 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.357+0000 7f27c4348700 1 -- 172.21.15.184:0/1055669738 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27bc0fafa0 con 0x7f27bc0f5410 2024-02-19T22:10:16.359 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.359+0000 7f27b2ffd700 1 -- 172.21.15.184:0/1055669738 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f27b800f340 con 0x7f27bc0f5410 2024-02-19T22:10:16.359 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:16.359 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:16.361 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.360+0000 7f27c4348700 1 -- 172.21.15.184:0/1055669738 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f27a802ff10 msgr2=0x7f27a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:16.361 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.360+0000 7f27c4348700 1 --2- 172.21.15.184:0/1055669738 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f27a802ff10 0x7f27a80323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f27ac0096a0 tx=0x7f27ac006b40).stop 2024-02-19T22:10:16.361 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.360+0000 7f27c4348700 1 -- 172.21.15.184:0/1055669738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27bc0f5410 msgr2=0x7f27bc109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:16.361 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.360+0000 7f27c4348700 1 --2- 172.21.15.184:0/1055669738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27bc0f5410 0x7f27bc109500 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f27b8005010 tx=0x7f27b800b040).stop 2024-02-19T22:10:16.361 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.360+0000 7f27c4348700 1 -- 172.21.15.184:0/1055669738 shutdown_connections 2024-02-19T22:10:16.361 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.360+0000 7f27c4348700 1 --2- 172.21.15.184:0/1055669738 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f27a802ff10 0x7f27a80323d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:16.361 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.360+0000 7f27c4348700 1 --2- 172.21.15.184:0/1055669738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27bc0f5410 0x7f27bc109500 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:16.361 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.360+0000 7f27c4348700 1 -- 172.21.15.184:0/1055669738 >> 172.21.15.184:0/1055669738 conn(0x7f27bc0f0b70 msgr2=0x7f27bc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:16.361 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.360+0000 7f27c4348700 1 -- 172.21.15.184:0/1055669738 shutdown_connections 2024-02-19T22:10:16.361 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:16.360+0000 7f27c4348700 1 -- 172.21.15.184:0/1055669738 wait complete. 2024-02-19T22:10:16.362 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:17.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:16 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:16.359174+0000 mon.smithi176 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.184:0/1055669738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:17.953 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:17.953 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:18.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:17 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:15.843622+0000 mgr.smithi176.faoogw (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:19.579 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.562+0000 7f05c27ed700 1 -- 172.21.15.184:0/643954159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05bc0f5410 msgr2=0x7f05bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.562+0000 7f05c27ed700 1 --2- 172.21.15.184:0/643954159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05bc0f5410 0x7f05bc0f5830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f05b0009fb0 tx=0x7f05b0009b30).stop 2024-02-19T22:10:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.562+0000 7f05c27ed700 1 -- 172.21.15.184:0/643954159 shutdown_connections 2024-02-19T22:10:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.562+0000 7f05c27ed700 1 --2- 172.21.15.184:0/643954159 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05bc0f5410 0x7f05bc0f5830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.562+0000 7f05c27ed700 1 -- 172.21.15.184:0/643954159 >> 172.21.15.184:0/643954159 conn(0x7f05bc0f0b70 msgr2=0x7f05bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.562+0000 7f05c27ed700 1 -- 172.21.15.184:0/643954159 shutdown_connections 2024-02-19T22:10:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.562+0000 7f05c27ed700 1 -- 172.21.15.184:0/643954159 wait complete. 2024-02-19T22:10:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.562+0000 7f05c27ed700 1 Processor -- start 2024-02-19T22:10:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.562+0000 7f05c27ed700 1 -- start start 2024-02-19T22:10:19.580 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.563+0000 7f05c27ed700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05bc0f5410 0x7f05bc109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:19.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.563+0000 7f05c27ed700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc060240 con 0x7f05bc0f5410 2024-02-19T22:10:19.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.563+0000 7f05bbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05bc0f5410 0x7f05bc109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:19.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.563+0000 7f05bbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05bc0f5410 0x7f05bc109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:54512/0 (socket says 172.21.15.184:54512) 2024-02-19T22:10:19.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.563+0000 7f05bbfff700 1 -- 172.21.15.184:0/2173253557 learned_addr learned my addr 172.21.15.184:0/2173253557 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:19.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.563+0000 7f05bbfff700 1 -- 172.21.15.184:0/2173253557 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b0005040 con 0x7f05bc0f5410 2024-02-19T22:10:19.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.563+0000 7f05bbfff700 1 --2- 172.21.15.184:0/2173253557 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05bc0f5410 0x7f05bc109510 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f05b0005010 tx=0x7f05b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:19.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.563+0000 7f05b8ff9700 1 -- 172.21.15.184:0/2173253557 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05b0009320 con 0x7f05bc0f5410 2024-02-19T22:10:19.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.563+0000 7f05b8ff9700 1 -- 172.21.15.184:0/2173253557 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05b0013380 con 0x7f05bc0f5410 2024-02-19T22:10:19.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.563+0000 7f05c27ed700 1 -- 172.21.15.184:0/2173253557 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc060440 con 0x7f05bc0f5410 2024-02-19T22:10:19.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.563+0000 7f05b8ff9700 1 -- 172.21.15.184:0/2173253557 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05b0014e50 con 0x7f05bc0f5410 2024-02-19T22:10:19.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.563+0000 7f05c27ed700 1 -- 172.21.15.184:0/2173253557 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc109bc0 con 0x7f05bc0f5410 2024-02-19T22:10:19.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.564+0000 7f05b8ff9700 1 -- 172.21.15.184:0/2173253557 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f05b00095f0 con 0x7f05bc0f5410 2024-02-19T22:10:19.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.564+0000 7f05c27ed700 1 -- 172.21.15.184:0/2173253557 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc045ad0 con 0x7f05bc0f5410 2024-02-19T22:10:19.581 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.564+0000 7f05b8ff9700 1 --2- 172.21.15.184:0/2173253557 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f05a402ff10 0x7f05a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:19.582 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.565+0000 7f05b8ff9700 1 -- 172.21.15.184:0/2173253557 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f05b0040e90 con 0x7f05bc0f5410 2024-02-19T22:10:19.582 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.565+0000 7f05bb7fe700 1 --2- 172.21.15.184:0/2173253557 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f05a402ff10 0x7f05a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:19.582 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.565+0000 7f05bb7fe700 1 --2- 172.21.15.184:0/2173253557 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f05a402ff10 0x7f05a40323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f05ac0096a0 tx=0x7f05ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:19.582 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.569+0000 7f05b8ff9700 1 -- 172.21.15.184:0/2173253557 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05b0014420 con 0x7f05bc0f5410 2024-02-19T22:10:19.775 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.774+0000 7f05c27ed700 1 -- 172.21.15.184:0/2173253557 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05bc10a3e0 con 0x7f05bc0f5410 2024-02-19T22:10:19.776 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.775+0000 7f05b8ff9700 1 -- 172.21.15.184:0/2173253557 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f05b0010230 con 0x7f05bc0f5410 2024-02-19T22:10:19.776 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:19.776 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:19.777 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.776+0000 7f05c27ed700 1 -- 172.21.15.184:0/2173253557 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f05a402ff10 msgr2=0x7f05a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:19.777 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.776+0000 7f05c27ed700 1 --2- 172.21.15.184:0/2173253557 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f05a402ff10 0x7f05a40323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f05ac0096a0 tx=0x7f05ac006b40).stop 2024-02-19T22:10:19.777 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.776+0000 7f05c27ed700 1 -- 172.21.15.184:0/2173253557 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05bc0f5410 msgr2=0x7f05bc109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:19.777 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.776+0000 7f05c27ed700 1 --2- 172.21.15.184:0/2173253557 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05bc0f5410 0x7f05bc109510 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f05b0005010 tx=0x7f05b000b040).stop 2024-02-19T22:10:19.777 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.776+0000 7f05c27ed700 1 -- 172.21.15.184:0/2173253557 shutdown_connections 2024-02-19T22:10:19.777 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.776+0000 7f05c27ed700 1 --2- 172.21.15.184:0/2173253557 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f05a402ff10 0x7f05a40323d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:19.778 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.776+0000 7f05c27ed700 1 --2- 172.21.15.184:0/2173253557 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f05bc0f5410 0x7f05bc109510 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:19.778 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.776+0000 7f05c27ed700 1 -- 172.21.15.184:0/2173253557 >> 172.21.15.184:0/2173253557 conn(0x7f05bc0f0b70 msgr2=0x7f05bc0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:19.778 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.776+0000 7f05c27ed700 1 -- 172.21.15.184:0/2173253557 shutdown_connections 2024-02-19T22:10:19.778 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:19.776+0000 7f05c27ed700 1 -- 172.21.15.184:0/2173253557 wait complete. 2024-02-19T22:10:19.778 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:20.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:17.843916+0000 mgr.smithi176.faoogw (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:21.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:20 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:19.776055+0000 mon.smithi176 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.184:0/2173253557' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:21.449 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:21.450 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:22.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:19.844244+0000 mgr.smithi176.faoogw (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:23.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.178+0000 7fa0b6861700 1 -- 172.21.15.184:0/3408402364 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b00f5c40 msgr2=0x7fa0b00f8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:23.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.178+0000 7fa0b6861700 1 --2- 172.21.15.184:0/3408402364 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b00f5c40 0x7fa0b00f8030 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fa0a0009fb0 tx=0x7fa0a0009b30).stop 2024-02-19T22:10:23.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.178+0000 7fa0b6861700 1 -- 172.21.15.184:0/3408402364 shutdown_connections 2024-02-19T22:10:23.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.178+0000 7fa0b6861700 1 --2- 172.21.15.184:0/3408402364 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b00f5c40 0x7fa0b00f8030 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:23.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.178+0000 7fa0b6861700 1 -- 172.21.15.184:0/3408402364 >> 172.21.15.184:0/3408402364 conn(0x7fa0b00efe40 msgr2=0x7fa0b00f2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:23.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.178+0000 7fa0b6861700 1 -- 172.21.15.184:0/3408402364 shutdown_connections 2024-02-19T22:10:23.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.179+0000 7fa0b6861700 1 -- 172.21.15.184:0/3408402364 wait complete. 2024-02-19T22:10:23.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.179+0000 7fa0b6861700 1 Processor -- start 2024-02-19T22:10:23.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.179+0000 7fa0b6861700 1 -- start start 2024-02-19T22:10:23.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.179+0000 7fa0b6861700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b00f5c40 0x7fa0b0106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:23.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.179+0000 7fa0b6861700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b0106d00 con 0x7fa0b00f5c40 2024-02-19T22:10:23.182 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.180+0000 7fa0affff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b00f5c40 0x7fa0b0106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:23.182 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.180+0000 7fa0affff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b00f5c40 0x7fa0b0106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:54524/0 (socket says 172.21.15.184:54524) 2024-02-19T22:10:23.182 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.180+0000 7fa0affff700 1 -- 172.21.15.184:0/758639522 learned_addr learned my addr 172.21.15.184:0/758639522 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:23.182 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.180+0000 7fa0affff700 1 -- 172.21.15.184:0/758639522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a0005040 con 0x7fa0b00f5c40 2024-02-19T22:10:23.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.180+0000 7fa0affff700 1 --2- 172.21.15.184:0/758639522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b00f5c40 0x7fa0b0106810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fa0a0009f80 tx=0x7fa0a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:23.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.180+0000 7fa0acff9700 1 -- 172.21.15.184:0/758639522 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0a0009320 con 0x7fa0b00f5c40 2024-02-19T22:10:23.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.180+0000 7fa0b6861700 1 -- 172.21.15.184:0/758639522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b0106f00 con 0x7fa0b00f5c40 2024-02-19T22:10:23.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.180+0000 7fa0acff9700 1 -- 172.21.15.184:0/758639522 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0a0013380 con 0x7fa0b00f5c40 2024-02-19T22:10:23.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.180+0000 7fa0acff9700 1 -- 172.21.15.184:0/758639522 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0a001eda0 con 0x7fa0b00f5c40 2024-02-19T22:10:23.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.180+0000 7fa0b6861700 1 -- 172.21.15.184:0/758639522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b0107390 con 0x7fa0b00f5c40 2024-02-19T22:10:23.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.181+0000 7fa0acff9700 1 -- 172.21.15.184:0/758639522 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0a0014420 con 0x7fa0b00f5c40 2024-02-19T22:10:23.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.181+0000 7fa0acff9700 1 --2- 172.21.15.184:0/758639522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa09802ffb0 0x7fa098032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:23.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.181+0000 7fa0acff9700 1 -- 172.21.15.184:0/758639522 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0a0040e80 con 0x7fa0b00f5c40 2024-02-19T22:10:23.184 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.181+0000 7fa0b6861700 1 -- 172.21.15.184:0/758639522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b0045ad0 con 0x7fa0b00f5c40 2024-02-19T22:10:23.184 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.182+0000 7fa0af7fe700 1 --2- 172.21.15.184:0/758639522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa09802ffb0 0x7fa098032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:23.184 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.182+0000 7fa0af7fe700 1 --2- 172.21.15.184:0/758639522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa09802ffb0 0x7fa098032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fa0a40096a0 tx=0x7fa0a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:23.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.187+0000 7fa0acff9700 1 -- 172.21.15.184:0/758639522 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0a0027490 con 0x7fa0b00f5c40 2024-02-19T22:10:23.393 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.392+0000 7fa0b6861700 1 -- 172.21.15.184:0/758639522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0b01080b0 con 0x7fa0b00f5c40 2024-02-19T22:10:23.394 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.394+0000 7fa0acff9700 1 -- 172.21.15.184:0/758639522 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa0a001c8f0 con 0x7fa0b00f5c40 2024-02-19T22:10:23.394 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:23.394 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:23.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.394+0000 7fa0b6861700 1 -- 172.21.15.184:0/758639522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa09802ffb0 msgr2=0x7fa098032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:23.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.394+0000 7fa0b6861700 1 --2- 172.21.15.184:0/758639522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa09802ffb0 0x7fa098032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fa0a40096a0 tx=0x7fa0a4006b40).stop 2024-02-19T22:10:23.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.394+0000 7fa0b6861700 1 -- 172.21.15.184:0/758639522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b00f5c40 msgr2=0x7fa0b0106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:23.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.394+0000 7fa0b6861700 1 --2- 172.21.15.184:0/758639522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b00f5c40 0x7fa0b0106810 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fa0a0009f80 tx=0x7fa0a000b040).stop 2024-02-19T22:10:23.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.394+0000 7fa0b6861700 1 -- 172.21.15.184:0/758639522 shutdown_connections 2024-02-19T22:10:23.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.394+0000 7fa0b6861700 1 --2- 172.21.15.184:0/758639522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa09802ffb0 0x7fa098032470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:23.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.394+0000 7fa0b6861700 1 --2- 172.21.15.184:0/758639522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa0b00f5c40 0x7fa0b0106810 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:23.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.394+0000 7fa0b6861700 1 -- 172.21.15.184:0/758639522 >> 172.21.15.184:0/758639522 conn(0x7fa0b00efe40 msgr2=0x7fa0b00f5200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:23.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.395+0000 7fa0b6861700 1 -- 172.21.15.184:0/758639522 shutdown_connections 2024-02-19T22:10:23.396 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:23.395+0000 7fa0b6861700 1 -- 172.21.15.184:0/758639522 wait complete. 2024-02-19T22:10:23.397 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:24.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:21.844623+0000 mgr.smithi176.faoogw (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:24.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:23.393951+0000 mon.smithi176 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.184:0/758639522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:25.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:25.043 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:26.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:23.844983+0000 mgr.smithi176.faoogw (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:26.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.738+0000 7f96dec5c700 1 -- 172.21.15.184:0/1058197983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96d80f6480 msgr2=0x7f96d80f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:26.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.738+0000 7f96dec5c700 1 --2- 172.21.15.184:0/1058197983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96d80f6480 0x7f96d80f68a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f96cc009fb0 tx=0x7f96cc009b30).stop 2024-02-19T22:10:26.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.738+0000 7f96dec5c700 1 -- 172.21.15.184:0/1058197983 shutdown_connections 2024-02-19T22:10:26.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.738+0000 7f96dec5c700 1 --2- 172.21.15.184:0/1058197983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96d80f6480 0x7f96d80f68a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:26.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.738+0000 7f96dec5c700 1 -- 172.21.15.184:0/1058197983 >> 172.21.15.184:0/1058197983 conn(0x7f96d80f1c00 msgr2=0x7f96d80f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:26.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.738+0000 7f96dec5c700 1 -- 172.21.15.184:0/1058197983 shutdown_connections 2024-02-19T22:10:26.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.738+0000 7f96dec5c700 1 -- 172.21.15.184:0/1058197983 wait complete. 2024-02-19T22:10:26.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.739+0000 7f96dec5c700 1 Processor -- start 2024-02-19T22:10:26.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.739+0000 7f96dec5c700 1 -- start start 2024-02-19T22:10:26.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.739+0000 7f96dec5c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96d80f6480 0x7f96d806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:26.739 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.739+0000 7f96dec5c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d8069e20 con 0x7f96d80f6480 2024-02-19T22:10:26.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.739+0000 7f96dc9f8700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96d80f6480 0x7f96d806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:26.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.739+0000 7f96dc9f8700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96d80f6480 0x7f96d806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:50684/0 (socket says 172.21.15.184:50684) 2024-02-19T22:10:26.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.739+0000 7f96dc9f8700 1 -- 172.21.15.184:0/718969102 learned_addr learned my addr 172.21.15.184:0/718969102 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:26.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.740+0000 7f96dc9f8700 1 -- 172.21.15.184:0/718969102 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc005040 con 0x7f96d80f6480 2024-02-19T22:10:26.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.740+0000 7f96dc9f8700 1 --2- 172.21.15.184:0/718969102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96d80f6480 0x7f96d806b9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f96cc005010 tx=0x7f96cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:26.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.740+0000 7f96d57fa700 1 -- 172.21.15.184:0/718969102 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96cc009320 con 0x7f96d80f6480 2024-02-19T22:10:26.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.740+0000 7f96dec5c700 1 -- 172.21.15.184:0/718969102 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d8069fc0 con 0x7f96d80f6480 2024-02-19T22:10:26.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.740+0000 7f96d57fa700 1 -- 172.21.15.184:0/718969102 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96cc013380 con 0x7f96d80f6480 2024-02-19T22:10:26.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.740+0000 7f96d57fa700 1 -- 172.21.15.184:0/718969102 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96cc014e50 con 0x7f96d80f6480 2024-02-19T22:10:26.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.740+0000 7f96dec5c700 1 -- 172.21.15.184:0/718969102 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d806a450 con 0x7f96d80f6480 2024-02-19T22:10:26.742 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.741+0000 7f96d57fa700 1 -- 172.21.15.184:0/718969102 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96cc014420 con 0x7f96d80f6480 2024-02-19T22:10:26.742 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.741+0000 7f96dec5c700 1 -- 172.21.15.184:0/718969102 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d8059100 con 0x7f96d80f6480 2024-02-19T22:10:26.742 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.741+0000 7f96d57fa700 1 --2- 172.21.15.184:0/718969102 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f96c802ff10 0x7f96c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:26.742 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.741+0000 7f96d57fa700 1 -- 172.21.15.184:0/718969102 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f96cc040e70 con 0x7f96d80f6480 2024-02-19T22:10:26.742 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.741+0000 7f96d7fff700 1 --2- 172.21.15.184:0/718969102 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f96c802ff10 0x7f96c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:26.742 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.742+0000 7f96d7fff700 1 --2- 172.21.15.184:0/718969102 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f96c802ff10 0x7f96c80323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f96c40096a0 tx=0x7f96c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:26.745 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.746+0000 7f96d57fa700 1 -- 172.21.15.184:0/718969102 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96cc0095f0 con 0x7f96d80f6480 2024-02-19T22:10:26.951 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.951+0000 7f96dec5c700 1 -- 172.21.15.184:0/718969102 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96d80fc090 con 0x7f96d80f6480 2024-02-19T22:10:26.951 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.952+0000 7f96d57fa700 1 -- 172.21.15.184:0/718969102 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f96cc01db70 con 0x7f96d80f6480 2024-02-19T22:10:26.952 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:26.952 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:26.953 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.953+0000 7f96dec5c700 1 -- 172.21.15.184:0/718969102 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f96c802ff10 msgr2=0x7f96c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:26.953 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.953+0000 7f96dec5c700 1 --2- 172.21.15.184:0/718969102 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f96c802ff10 0x7f96c80323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f96c40096a0 tx=0x7f96c4006b40).stop 2024-02-19T22:10:26.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.953+0000 7f96dec5c700 1 -- 172.21.15.184:0/718969102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96d80f6480 msgr2=0x7f96d806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:26.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.953+0000 7f96dec5c700 1 --2- 172.21.15.184:0/718969102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96d80f6480 0x7f96d806b9a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f96cc005010 tx=0x7f96cc00b040).stop 2024-02-19T22:10:26.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.953+0000 7f96dec5c700 1 -- 172.21.15.184:0/718969102 shutdown_connections 2024-02-19T22:10:26.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.953+0000 7f96dec5c700 1 --2- 172.21.15.184:0/718969102 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f96c802ff10 0x7f96c80323d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:26.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.953+0000 7f96dec5c700 1 --2- 172.21.15.184:0/718969102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f96d80f6480 0x7f96d806b9a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:26.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.953+0000 7f96dec5c700 1 -- 172.21.15.184:0/718969102 >> 172.21.15.184:0/718969102 conn(0x7f96d80f1c00 msgr2=0x7f96d80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:26.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.953+0000 7f96dec5c700 1 -- 172.21.15.184:0/718969102 shutdown_connections 2024-02-19T22:10:26.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:26.953+0000 7f96dec5c700 1 -- 172.21.15.184:0/718969102 wait complete. 2024-02-19T22:10:26.954 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:28.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:25.845339+0000 mgr.smithi176.faoogw (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:28.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:26.951924+0000 mon.smithi176 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.184:0/718969102' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:28.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:28.538 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:30.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:29 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:27.845678+0000 mgr.smithi176.faoogw (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:30.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.319+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/3157848117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8f00f53f0 msgr2=0x7fe8f00f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:30.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.319+0000 7fe8f7d5f700 1 --2- 172.21.15.184:0/3157848117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8f00f53f0 0x7fe8f00f5810 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fe8e0009fb0 tx=0x7fe8e0009b30).stop 2024-02-19T22:10:30.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.319+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/3157848117 shutdown_connections 2024-02-19T22:10:30.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.319+0000 7fe8f7d5f700 1 --2- 172.21.15.184:0/3157848117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8f00f53f0 0x7fe8f00f5810 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:30.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.319+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/3157848117 >> 172.21.15.184:0/3157848117 conn(0x7fe8f00f0b70 msgr2=0x7fe8f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:30.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.320+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/3157848117 shutdown_connections 2024-02-19T22:10:30.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.320+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/3157848117 wait complete. 2024-02-19T22:10:30.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.320+0000 7fe8f7d5f700 1 Processor -- start 2024-02-19T22:10:30.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.320+0000 7fe8f7d5f700 1 -- start start 2024-02-19T22:10:30.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.320+0000 7fe8f7d5f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8f00f53f0 0x7fe8f01094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:30.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.320+0000 7fe8f7d5f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f0060220 con 0x7fe8f00f53f0 2024-02-19T22:10:30.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.321+0000 7fe8f5afb700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8f00f53f0 0x7fe8f01094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:30.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.321+0000 7fe8f5afb700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8f00f53f0 0x7fe8f01094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:50704/0 (socket says 172.21.15.184:50704) 2024-02-19T22:10:30.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.321+0000 7fe8f5afb700 1 -- 172.21.15.184:0/810789628 learned_addr learned my addr 172.21.15.184:0/810789628 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:30.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.321+0000 7fe8f5afb700 1 -- 172.21.15.184:0/810789628 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e0005040 con 0x7fe8f00f53f0 2024-02-19T22:10:30.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.321+0000 7fe8f5afb700 1 --2- 172.21.15.184:0/810789628 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8f00f53f0 0x7fe8f01094e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe8e00089f0 tx=0x7fe8e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:30.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.321+0000 7fe8e67fc700 1 -- 172.21.15.184:0/810789628 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e0009320 con 0x7fe8f00f53f0 2024-02-19T22:10:30.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.321+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/810789628 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f0060420 con 0x7fe8f00f53f0 2024-02-19T22:10:30.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.321+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/810789628 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f0109b90 con 0x7fe8f00f53f0 2024-02-19T22:10:30.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.321+0000 7fe8e67fc700 1 -- 172.21.15.184:0/810789628 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8e0013380 con 0x7fe8f00f53f0 2024-02-19T22:10:30.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.321+0000 7fe8e67fc700 1 -- 172.21.15.184:0/810789628 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8e001ed80 con 0x7fe8f00f53f0 2024-02-19T22:10:30.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.322+0000 7fe8e67fc700 1 -- 172.21.15.184:0/810789628 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8e001e420 con 0x7fe8f00f53f0 2024-02-19T22:10:30.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.322+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/810789628 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f0059100 con 0x7fe8f00f53f0 2024-02-19T22:10:30.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.322+0000 7fe8e67fc700 1 --2- 172.21.15.184:0/810789628 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe8dc02ff60 0x7fe8dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:30.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.322+0000 7fe8e67fc700 1 -- 172.21.15.184:0/810789628 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8e00411e0 con 0x7fe8f00f53f0 2024-02-19T22:10:30.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.322+0000 7fe8f52fa700 1 --2- 172.21.15.184:0/810789628 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe8dc02ff60 0x7fe8dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:30.324 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.323+0000 7fe8f52fa700 1 --2- 172.21.15.184:0/810789628 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe8dc02ff60 0x7fe8dc032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe8ec0096a0 tx=0x7fe8ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:30.327 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.327+0000 7fe8e67fc700 1 -- 172.21.15.184:0/810789628 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8e000fb30 con 0x7fe8f00f53f0 2024-02-19T22:10:30.532 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.531+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/810789628 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8f0045ad0 con 0x7fe8f00f53f0 2024-02-19T22:10:30.533 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.532+0000 7fe8e67fc700 1 -- 172.21.15.184:0/810789628 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe8e002c2d0 con 0x7fe8f00f53f0 2024-02-19T22:10:30.533 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:30.533 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:30.534 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.533+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/810789628 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe8dc02ff60 msgr2=0x7fe8dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:30.534 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.533+0000 7fe8f7d5f700 1 --2- 172.21.15.184:0/810789628 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe8dc02ff60 0x7fe8dc032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe8ec0096a0 tx=0x7fe8ec006b40).stop 2024-02-19T22:10:30.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.533+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/810789628 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8f00f53f0 msgr2=0x7fe8f01094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:30.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.533+0000 7fe8f7d5f700 1 --2- 172.21.15.184:0/810789628 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8f00f53f0 0x7fe8f01094e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe8e00089f0 tx=0x7fe8e000b040).stop 2024-02-19T22:10:30.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.533+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/810789628 shutdown_connections 2024-02-19T22:10:30.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.533+0000 7fe8f7d5f700 1 --2- 172.21.15.184:0/810789628 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe8dc02ff60 0x7fe8dc032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:30.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.533+0000 7fe8f7d5f700 1 --2- 172.21.15.184:0/810789628 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8f00f53f0 0x7fe8f01094e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:30.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.533+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/810789628 >> 172.21.15.184:0/810789628 conn(0x7fe8f00f0b70 msgr2=0x7fe8f00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:30.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.533+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/810789628 shutdown_connections 2024-02-19T22:10:30.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:30.534+0000 7fe8f7d5f700 1 -- 172.21.15.184:0/810789628 wait complete. 2024-02-19T22:10:30.535 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:31.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:30 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:29.845916+0000 mgr.smithi176.faoogw (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:31.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:30 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:30.533238+0000 mon.smithi176 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.184:0/810789628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:32.132 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:32.132 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:33.384 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:32 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:31.846180+0000 mgr.smithi176.faoogw (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:33.973 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.951+0000 7efd8bfff700 1 -- 172.21.15.184:0/1898006082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd8c0f62a0 msgr2=0x7efd8c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:33.974 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.951+0000 7efd8bfff700 1 --2- 172.21.15.184:0/1898006082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd8c0f62a0 0x7efd8c0f66c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7efd7c009fb0 tx=0x7efd7c009b30).stop 2024-02-19T22:10:33.974 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.951+0000 7efd8bfff700 1 -- 172.21.15.184:0/1898006082 shutdown_connections 2024-02-19T22:10:33.974 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.951+0000 7efd8bfff700 1 --2- 172.21.15.184:0/1898006082 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd8c0f62a0 0x7efd8c0f66c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:33.974 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.951+0000 7efd8bfff700 1 -- 172.21.15.184:0/1898006082 >> 172.21.15.184:0/1898006082 conn(0x7efd8c0f1aa0 msgr2=0x7efd8c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:33.974 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.951+0000 7efd8bfff700 1 -- 172.21.15.184:0/1898006082 shutdown_connections 2024-02-19T22:10:33.974 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.951+0000 7efd8bfff700 1 -- 172.21.15.184:0/1898006082 wait complete. 2024-02-19T22:10:33.974 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.951+0000 7efd8bfff700 1 Processor -- start 2024-02-19T22:10:33.974 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.951+0000 7efd8bfff700 1 -- start start 2024-02-19T22:10:33.974 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.951+0000 7efd8bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd8c0f62a0 0x7efd8c1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:33.974 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.951+0000 7efd8bfff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c106ac0 con 0x7efd8c0f62a0 2024-02-19T22:10:33.974 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.952+0000 7efd8affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd8c0f62a0 0x7efd8c1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:33.974 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.952+0000 7efd8affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd8c0f62a0 0x7efd8c1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:50712/0 (socket says 172.21.15.184:50712) 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.952+0000 7efd8affd700 1 -- 172.21.15.184:0/1827351164 learned_addr learned my addr 172.21.15.184:0/1827351164 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.952+0000 7efd8affd700 1 -- 172.21.15.184:0/1827351164 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c005040 con 0x7efd8c0f62a0 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.952+0000 7efd8affd700 1 --2- 172.21.15.184:0/1827351164 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd8c0f62a0 0x7efd8c1065d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7efd7c0089f0 tx=0x7efd7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.952+0000 7efd737fe700 1 -- 172.21.15.184:0/1827351164 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd7c009320 con 0x7efd8c0f62a0 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.952+0000 7efd8bfff700 1 -- 172.21.15.184:0/1827351164 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd8c106cc0 con 0x7efd8c0f62a0 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.952+0000 7efd737fe700 1 -- 172.21.15.184:0/1827351164 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd7c013380 con 0x7efd8c0f62a0 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.952+0000 7efd737fe700 1 -- 172.21.15.184:0/1827351164 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd7c014e20 con 0x7efd8c0f62a0 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.952+0000 7efd8bfff700 1 -- 172.21.15.184:0/1827351164 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd8c107150 con 0x7efd8c0f62a0 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.953+0000 7efd737fe700 1 -- 172.21.15.184:0/1827351164 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efd7c014420 con 0x7efd8c0f62a0 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.953+0000 7efd737fe700 1 --2- 172.21.15.184:0/1827351164 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7efd7402ff10 0x7efd740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.953+0000 7efd737fe700 1 -- 172.21.15.184:0/1827351164 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efd7c040fc0 con 0x7efd8c0f62a0 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.953+0000 7efd8bfff700 1 -- 172.21.15.184:0/1827351164 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd8c045ad0 con 0x7efd8c0f62a0 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.954+0000 7efd8a7fc700 1 --2- 172.21.15.184:0/1827351164 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7efd7402ff10 0x7efd740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.954+0000 7efd8a7fc700 1 --2- 172.21.15.184:0/1827351164 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7efd7402ff10 0x7efd740323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7efd800096a0 tx=0x7efd80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:33.975 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:33.958+0000 7efd737fe700 1 -- 172.21.15.184:0/1827351164 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd7c0095f0 con 0x7efd8c0f62a0 2024-02-19T22:10:34.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:34.164+0000 7efd8bfff700 1 -- 172.21.15.184:0/1827351164 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd8c108000 con 0x7efd8c0f62a0 2024-02-19T22:10:34.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:34.165+0000 7efd737fe700 1 -- 172.21.15.184:0/1827351164 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efd7c01db70 con 0x7efd8c0f62a0 2024-02-19T22:10:34.166 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:34.166 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:34.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:34.167+0000 7efd8bfff700 1 -- 172.21.15.184:0/1827351164 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7efd7402ff10 msgr2=0x7efd740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:34.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:34.167+0000 7efd8bfff700 1 --2- 172.21.15.184:0/1827351164 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7efd7402ff10 0x7efd740323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7efd800096a0 tx=0x7efd80006b40).stop 2024-02-19T22:10:34.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:34.167+0000 7efd8bfff700 1 -- 172.21.15.184:0/1827351164 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd8c0f62a0 msgr2=0x7efd8c1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:34.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:34.167+0000 7efd8bfff700 1 --2- 172.21.15.184:0/1827351164 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd8c0f62a0 0x7efd8c1065d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7efd7c0089f0 tx=0x7efd7c00b040).stop 2024-02-19T22:10:34.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:34.167+0000 7efd8bfff700 1 -- 172.21.15.184:0/1827351164 shutdown_connections 2024-02-19T22:10:34.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:34.167+0000 7efd8bfff700 1 --2- 172.21.15.184:0/1827351164 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7efd7402ff10 0x7efd740323d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:34.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:34.167+0000 7efd8bfff700 1 --2- 172.21.15.184:0/1827351164 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd8c0f62a0 0x7efd8c1065d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:34.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:34.167+0000 7efd8bfff700 1 -- 172.21.15.184:0/1827351164 >> 172.21.15.184:0/1827351164 conn(0x7efd8c0f1aa0 msgr2=0x7efd8c0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:34.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:34.167+0000 7efd8bfff700 1 -- 172.21.15.184:0/1827351164 shutdown_connections 2024-02-19T22:10:34.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:34.167+0000 7efd8bfff700 1 -- 172.21.15.184:0/1827351164 wait complete. 2024-02-19T22:10:34.169 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:35.384 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:34 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:33.846487+0000 mgr.smithi176.faoogw (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:35.385 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:34 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:34.166437+0000 mon.smithi176 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.184:0/1827351164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:35.806 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:35.806 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:37.384 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:35.846835+0000 mgr.smithi176.faoogw (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:37.385 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:35.967756+0000 mon.smithi176 (mon.0) 141 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:10:37.385 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:35.970581+0000 mon.smithi176 (mon.0) 142 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:10:37.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.522+0000 7f59bab9d700 1 -- 172.21.15.184:0/1949591664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b40f51a0 msgr2=0x7f59b40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.522+0000 7f59bab9d700 1 --2- 172.21.15.184:0/1949591664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b40f51a0 0x7f59b40f55c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f59a4009fb0 tx=0x7f59a4009b30).stop 2024-02-19T22:10:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.522+0000 7f59bab9d700 1 -- 172.21.15.184:0/1949591664 shutdown_connections 2024-02-19T22:10:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.522+0000 7f59bab9d700 1 --2- 172.21.15.184:0/1949591664 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b40f51a0 0x7f59b40f55c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.522+0000 7f59bab9d700 1 -- 172.21.15.184:0/1949591664 >> 172.21.15.184:0/1949591664 conn(0x7f59b40f09a0 msgr2=0x7f59b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.522+0000 7f59bab9d700 1 -- 172.21.15.184:0/1949591664 shutdown_connections 2024-02-19T22:10:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.522+0000 7f59bab9d700 1 -- 172.21.15.184:0/1949591664 wait complete. 2024-02-19T22:10:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.523+0000 7f59bab9d700 1 Processor -- start 2024-02-19T22:10:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.523+0000 7f59bab9d700 1 -- start start 2024-02-19T22:10:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.523+0000 7f59bab9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b40f51a0 0x7f59b41092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.523+0000 7f59bab9d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b4060220 con 0x7f59b40f51a0 2024-02-19T22:10:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.523+0000 7f59b9b9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b40f51a0 0x7f59b41092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.523+0000 7f59b9b9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b40f51a0 0x7f59b41092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:50888/0 (socket says 172.21.15.184:50888) 2024-02-19T22:10:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.523+0000 7f59b9b9b700 1 -- 172.21.15.184:0/2256220271 learned_addr learned my addr 172.21.15.184:0/2256220271 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.523+0000 7f59b9b9b700 1 -- 172.21.15.184:0/2256220271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a4005040 con 0x7f59b40f51a0 2024-02-19T22:10:37.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.523+0000 7f59b9b9b700 1 --2- 172.21.15.184:0/2256220271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b40f51a0 0x7f59b41092f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f59a40089f0 tx=0x7f59a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:37.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.524+0000 7f59aa7fc700 1 -- 172.21.15.184:0/2256220271 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59a4009320 con 0x7f59b40f51a0 2024-02-19T22:10:37.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.524+0000 7f59bab9d700 1 -- 172.21.15.184:0/2256220271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b4060420 con 0x7f59b40f51a0 2024-02-19T22:10:37.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.524+0000 7f59aa7fc700 1 -- 172.21.15.184:0/2256220271 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59a4013380 con 0x7f59b40f51a0 2024-02-19T22:10:37.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.524+0000 7f59aa7fc700 1 -- 172.21.15.184:0/2256220271 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59a4014e20 con 0x7f59b40f51a0 2024-02-19T22:10:37.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.524+0000 7f59bab9d700 1 -- 172.21.15.184:0/2256220271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b41099a0 con 0x7f59b40f51a0 2024-02-19T22:10:37.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.524+0000 7f59aa7fc700 1 -- 172.21.15.184:0/2256220271 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59a40263a0 con 0x7f59b40f51a0 2024-02-19T22:10:37.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.525+0000 7f59aa7fc700 1 --2- 172.21.15.184:0/2256220271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f59a002ff10 0x7f59a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:37.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.525+0000 7f59aa7fc700 1 -- 172.21.15.184:0/2256220271 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59a4040fa0 con 0x7f59b40f51a0 2024-02-19T22:10:37.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.525+0000 7f59bab9d700 1 -- 172.21.15.184:0/2256220271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b4059100 con 0x7f59b40f51a0 2024-02-19T22:10:37.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.525+0000 7f59b939a700 1 --2- 172.21.15.184:0/2256220271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f59a002ff10 0x7f59a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:37.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.525+0000 7f59b939a700 1 --2- 172.21.15.184:0/2256220271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f59a002ff10 0x7f59a00323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f59b00096a0 tx=0x7f59b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:37.530 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.529+0000 7f59aa7fc700 1 -- 172.21.15.184:0/2256220271 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59a4012920 con 0x7f59b40f51a0 2024-02-19T22:10:37.735 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.734+0000 7f59bab9d700 1 -- 172.21.15.184:0/2256220271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59b4045ad0 con 0x7f59b40f51a0 2024-02-19T22:10:37.736 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.735+0000 7f59aa7fc700 1 -- 172.21.15.184:0/2256220271 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f59b4045ad0 con 0x7f59b40f51a0 2024-02-19T22:10:37.736 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:37.736 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:37.738 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.737+0000 7f59bab9d700 1 -- 172.21.15.184:0/2256220271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f59a002ff10 msgr2=0x7f59a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:37.738 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.737+0000 7f59bab9d700 1 --2- 172.21.15.184:0/2256220271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f59a002ff10 0x7f59a00323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f59b00096a0 tx=0x7f59b0006b40).stop 2024-02-19T22:10:37.738 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.737+0000 7f59bab9d700 1 -- 172.21.15.184:0/2256220271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b40f51a0 msgr2=0x7f59b41092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:37.738 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.737+0000 7f59bab9d700 1 --2- 172.21.15.184:0/2256220271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b40f51a0 0x7f59b41092f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f59a40089f0 tx=0x7f59a400b040).stop 2024-02-19T22:10:37.738 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.737+0000 7f59bab9d700 1 -- 172.21.15.184:0/2256220271 shutdown_connections 2024-02-19T22:10:37.738 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.737+0000 7f59bab9d700 1 --2- 172.21.15.184:0/2256220271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f59a002ff10 0x7f59a00323d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:37.738 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.737+0000 7f59bab9d700 1 --2- 172.21.15.184:0/2256220271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b40f51a0 0x7f59b41092f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:37.738 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.737+0000 7f59bab9d700 1 -- 172.21.15.184:0/2256220271 >> 172.21.15.184:0/2256220271 conn(0x7f59b40f09a0 msgr2=0x7f59b40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:37.738 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.737+0000 7f59bab9d700 1 -- 172.21.15.184:0/2256220271 shutdown_connections 2024-02-19T22:10:37.738 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:37.737+0000 7f59bab9d700 1 -- 172.21.15.184:0/2256220271 wait complete. 2024-02-19T22:10:37.739 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:38.384 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:37.736298+0000 mon.smithi176 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.184:0/2256220271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:39.313 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:39.314 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:39.384 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:38 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:37.847116+0000 mgr.smithi176.faoogw (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:41.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.139+0000 7f45b4809700 1 -- 172.21.15.184:0/1200953275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00f5950 msgr2=0x7f45b00f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:41.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.139+0000 7f45b4809700 1 --2- 172.21.15.184:0/1200953275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00f5950 0x7f45b00f5d70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f45a0009fb0 tx=0x7f45a0009b30).stop 2024-02-19T22:10:41.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.140+0000 7f45b4809700 1 -- 172.21.15.184:0/1200953275 shutdown_connections 2024-02-19T22:10:41.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.140+0000 7f45b4809700 1 --2- 172.21.15.184:0/1200953275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00f5950 0x7f45b00f5d70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:41.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.140+0000 7f45b4809700 1 -- 172.21.15.184:0/1200953275 >> 172.21.15.184:0/1200953275 conn(0x7f45b00f0260 msgr2=0x7f45b00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:41.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.140+0000 7f45b4809700 1 -- 172.21.15.184:0/1200953275 shutdown_connections 2024-02-19T22:10:41.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.140+0000 7f45b4809700 1 -- 172.21.15.184:0/1200953275 wait complete. 2024-02-19T22:10:41.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.140+0000 7f45b4809700 1 Processor -- start 2024-02-19T22:10:41.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.140+0000 7f45b4809700 1 -- start start 2024-02-19T22:10:41.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.140+0000 7f45b4809700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00f5950 0x7f45b00fd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:41.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.140+0000 7f45b4809700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b00fb940 con 0x7f45b00f5950 2024-02-19T22:10:41.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.140+0000 7f45ae59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00f5950 0x7f45b00fd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:41.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.140+0000 7f45ae59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00f5950 0x7f45b00fd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:50914/0 (socket says 172.21.15.184:50914) 2024-02-19T22:10:41.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.140+0000 7f45ae59c700 1 -- 172.21.15.184:0/2213495969 learned_addr learned my addr 172.21.15.184:0/2213495969 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:41.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.141+0000 7f45ae59c700 1 -- 172.21.15.184:0/2213495969 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a0005040 con 0x7f45b00f5950 2024-02-19T22:10:41.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.141+0000 7f45ae59c700 1 --2- 172.21.15.184:0/2213495969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00f5950 0x7f45b00fd4c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f45a0005010 tx=0x7f45a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:41.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.141+0000 7f459effd700 1 -- 172.21.15.184:0/2213495969 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45a0009320 con 0x7f45b00f5950 2024-02-19T22:10:41.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.141+0000 7f459effd700 1 -- 172.21.15.184:0/2213495969 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45a0013380 con 0x7f45b00f5950 2024-02-19T22:10:41.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.141+0000 7f45b4809700 1 -- 172.21.15.184:0/2213495969 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b00fbae0 con 0x7f45b00f5950 2024-02-19T22:10:41.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.141+0000 7f459effd700 1 -- 172.21.15.184:0/2213495969 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45a0014e50 con 0x7f45b00f5950 2024-02-19T22:10:41.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.141+0000 7f45b4809700 1 -- 172.21.15.184:0/2213495969 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b00fbf70 con 0x7f45b00f5950 2024-02-19T22:10:41.144 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.142+0000 7f459effd700 1 -- 172.21.15.184:0/2213495969 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45a0014420 con 0x7f45b00f5950 2024-02-19T22:10:41.144 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.142+0000 7f45b4809700 1 -- 172.21.15.184:0/2213495969 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b0059100 con 0x7f45b00f5950 2024-02-19T22:10:41.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.142+0000 7f459effd700 1 --2- 172.21.15.184:0/2213495969 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f459802ff10 0x7f45980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:41.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.142+0000 7f459effd700 1 -- 172.21.15.184:0/2213495969 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45a0040e60 con 0x7f45b00f5950 2024-02-19T22:10:41.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.142+0000 7f45add9b700 1 --2- 172.21.15.184:0/2213495969 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f459802ff10 0x7f45980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:41.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.142+0000 7f45add9b700 1 --2- 172.21.15.184:0/2213495969 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f459802ff10 0x7f45980323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f45a40096a0 tx=0x7f45a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:41.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.147+0000 7f459effd700 1 -- 172.21.15.184:0/2213495969 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45a00095f0 con 0x7f45b00f5950 2024-02-19T22:10:41.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.352+0000 7f45b4809700 1 -- 172.21.15.184:0/2213495969 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45b00faf80 con 0x7f45b00f5950 2024-02-19T22:10:41.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.352+0000 7f459effd700 1 -- 172.21.15.184:0/2213495969 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f45a001db70 con 0x7f45b00f5950 2024-02-19T22:10:41.353 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:41.353 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:41.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.354+0000 7f45b4809700 1 -- 172.21.15.184:0/2213495969 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f459802ff10 msgr2=0x7f45980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:41.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.354+0000 7f45b4809700 1 --2- 172.21.15.184:0/2213495969 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f459802ff10 0x7f45980323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f45a40096a0 tx=0x7f45a4006b40).stop 2024-02-19T22:10:41.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.354+0000 7f45b4809700 1 -- 172.21.15.184:0/2213495969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00f5950 msgr2=0x7f45b00fd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:41.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.354+0000 7f45b4809700 1 --2- 172.21.15.184:0/2213495969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00f5950 0x7f45b00fd4c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f45a0005010 tx=0x7f45a000b040).stop 2024-02-19T22:10:41.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.354+0000 7f45b4809700 1 -- 172.21.15.184:0/2213495969 shutdown_connections 2024-02-19T22:10:41.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.354+0000 7f45b4809700 1 --2- 172.21.15.184:0/2213495969 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f459802ff10 0x7f45980323d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:41.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.354+0000 7f45b4809700 1 --2- 172.21.15.184:0/2213495969 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00f5950 0x7f45b00fd4c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:41.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.354+0000 7f45b4809700 1 -- 172.21.15.184:0/2213495969 >> 172.21.15.184:0/2213495969 conn(0x7f45b00f0260 msgr2=0x7f45b00f4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:41.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.354+0000 7f45b4809700 1 -- 172.21.15.184:0/2213495969 shutdown_connections 2024-02-19T22:10:41.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:41.354+0000 7f45b4809700 1 -- 172.21.15.184:0/2213495969 wait complete. 2024-02-19T22:10:41.356 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:42.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:39.847356+0000 mgr.smithi176.faoogw (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:42.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:41.353596+0000 mon.smithi176 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.184:0/2213495969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:42.952 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:42.952 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:44.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:41.847684+0000 mgr.smithi176.faoogw (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:44.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.715+0000 7feea26ef700 1 -- 172.21.15.184:0/2882407553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee9c0f51a0 msgr2=0x7fee9c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:44.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.715+0000 7feea26ef700 1 --2- 172.21.15.184:0/2882407553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee9c0f51a0 0x7fee9c0f55c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fee98009fb0 tx=0x7fee98009b30).stop 2024-02-19T22:10:44.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.716+0000 7feea26ef700 1 -- 172.21.15.184:0/2882407553 shutdown_connections 2024-02-19T22:10:44.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.716+0000 7feea26ef700 1 --2- 172.21.15.184:0/2882407553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee9c0f51a0 0x7fee9c0f55c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.716+0000 7feea26ef700 1 -- 172.21.15.184:0/2882407553 >> 172.21.15.184:0/2882407553 conn(0x7fee9c0f09a0 msgr2=0x7fee9c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.716+0000 7feea26ef700 1 -- 172.21.15.184:0/2882407553 shutdown_connections 2024-02-19T22:10:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.716+0000 7feea26ef700 1 -- 172.21.15.184:0/2882407553 wait complete. 2024-02-19T22:10:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.716+0000 7feea26ef700 1 Processor -- start 2024-02-19T22:10:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.716+0000 7feea26ef700 1 -- start start 2024-02-19T22:10:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.716+0000 7feea26ef700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee9c0f51a0 0x7fee9c1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.716+0000 7feea26ef700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c106ac0 con 0x7fee9c0f51a0 2024-02-19T22:10:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.717+0000 7feea16ed700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee9c0f51a0 0x7fee9c1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.717+0000 7feea16ed700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee9c0f51a0 0x7fee9c1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:50934/0 (socket says 172.21.15.184:50934) 2024-02-19T22:10:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.717+0000 7feea16ed700 1 -- 172.21.15.184:0/2728918439 learned_addr learned my addr 172.21.15.184:0/2728918439 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.717+0000 7feea16ed700 1 -- 172.21.15.184:0/2728918439 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee98005040 con 0x7fee9c0f51a0 2024-02-19T22:10:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.717+0000 7feea16ed700 1 --2- 172.21.15.184:0/2728918439 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee9c0f51a0 0x7fee9c1065d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fee980089f0 tx=0x7fee9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.717+0000 7fee91ffb700 1 -- 172.21.15.184:0/2728918439 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee98009320 con 0x7fee9c0f51a0 2024-02-19T22:10:44.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.717+0000 7feea26ef700 1 -- 172.21.15.184:0/2728918439 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee9c106cc0 con 0x7fee9c0f51a0 2024-02-19T22:10:44.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.717+0000 7fee91ffb700 1 -- 172.21.15.184:0/2728918439 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee98013380 con 0x7fee9c0f51a0 2024-02-19T22:10:44.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.717+0000 7fee91ffb700 1 -- 172.21.15.184:0/2728918439 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee98014e20 con 0x7fee9c0f51a0 2024-02-19T22:10:44.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.718+0000 7feea26ef700 1 -- 172.21.15.184:0/2728918439 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee9c107150 con 0x7fee9c0f51a0 2024-02-19T22:10:44.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.718+0000 7fee91ffb700 1 -- 172.21.15.184:0/2728918439 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee980095f0 con 0x7fee9c0f51a0 2024-02-19T22:10:44.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.718+0000 7feea26ef700 1 -- 172.21.15.184:0/2728918439 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee9c045ad0 con 0x7fee9c0f51a0 2024-02-19T22:10:44.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.718+0000 7fee91ffb700 1 --2- 172.21.15.184:0/2728918439 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fee8802ff10 0x7fee880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:44.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.719+0000 7fee91ffb700 1 -- 172.21.15.184:0/2728918439 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee98040f80 con 0x7fee9c0f51a0 2024-02-19T22:10:44.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.719+0000 7feea0eec700 1 --2- 172.21.15.184:0/2728918439 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fee8802ff10 0x7fee880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:44.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.719+0000 7feea0eec700 1 --2- 172.21.15.184:0/2728918439 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fee8802ff10 0x7fee880323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fee8c0096a0 tx=0x7fee8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:44.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.723+0000 7fee91ffb700 1 -- 172.21.15.184:0/2728918439 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee98014420 con 0x7fee9c0f51a0 2024-02-19T22:10:44.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.929+0000 7feea26ef700 1 -- 172.21.15.184:0/2728918439 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee9c108770 con 0x7fee9c0f51a0 2024-02-19T22:10:44.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.929+0000 7fee91ffb700 1 -- 172.21.15.184:0/2728918439 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fee98010230 con 0x7fee9c0f51a0 2024-02-19T22:10:44.930 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:44.931 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:44.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.931+0000 7feea26ef700 1 -- 172.21.15.184:0/2728918439 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fee8802ff10 msgr2=0x7fee880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:44.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.931+0000 7feea26ef700 1 --2- 172.21.15.184:0/2728918439 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fee8802ff10 0x7fee880323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fee8c0096a0 tx=0x7fee8c006b40).stop 2024-02-19T22:10:44.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.931+0000 7feea26ef700 1 -- 172.21.15.184:0/2728918439 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee9c0f51a0 msgr2=0x7fee9c1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:44.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.931+0000 7feea26ef700 1 --2- 172.21.15.184:0/2728918439 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee9c0f51a0 0x7fee9c1065d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fee980089f0 tx=0x7fee9800b040).stop 2024-02-19T22:10:44.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.931+0000 7feea26ef700 1 -- 172.21.15.184:0/2728918439 shutdown_connections 2024-02-19T22:10:44.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.931+0000 7feea26ef700 1 --2- 172.21.15.184:0/2728918439 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fee8802ff10 0x7fee880323d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:44.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.931+0000 7feea26ef700 1 --2- 172.21.15.184:0/2728918439 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee9c0f51a0 0x7fee9c1065d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:44.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.931+0000 7feea26ef700 1 -- 172.21.15.184:0/2728918439 >> 172.21.15.184:0/2728918439 conn(0x7fee9c0f09a0 msgr2=0x7fee9c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:44.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.931+0000 7feea26ef700 1 -- 172.21.15.184:0/2728918439 shutdown_connections 2024-02-19T22:10:44.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:44.931+0000 7feea26ef700 1 -- 172.21.15.184:0/2728918439 wait complete. 2024-02-19T22:10:44.933 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:46.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:43.847933+0000 mgr.smithi176.faoogw (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:46.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:44.930695+0000 mon.smithi176 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.184:0/2728918439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:46.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:46.510 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:48.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:45.848187+0000 mgr.smithi176.faoogw (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.194+0000 7f419b8a1700 1 -- 172.21.15.184:0/4014076526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940f5410 msgr2=0x7f41940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.194+0000 7f419b8a1700 1 --2- 172.21.15.184:0/4014076526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940f5410 0x7f41940f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f4184009fb0 tx=0x7f4184009b30).stop 2024-02-19T22:10:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.195+0000 7f419b8a1700 1 -- 172.21.15.184:0/4014076526 shutdown_connections 2024-02-19T22:10:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.195+0000 7f419b8a1700 1 --2- 172.21.15.184:0/4014076526 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940f5410 0x7f41940f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.195+0000 7f419b8a1700 1 -- 172.21.15.184:0/4014076526 >> 172.21.15.184:0/4014076526 conn(0x7f41940f0b70 msgr2=0x7f41940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.195+0000 7f419b8a1700 1 -- 172.21.15.184:0/4014076526 shutdown_connections 2024-02-19T22:10:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.195+0000 7f419b8a1700 1 -- 172.21.15.184:0/4014076526 wait complete. 2024-02-19T22:10:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.195+0000 7f419b8a1700 1 Processor -- start 2024-02-19T22:10:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.195+0000 7f419b8a1700 1 -- start start 2024-02-19T22:10:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.195+0000 7f419b8a1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940f5410 0x7f41941067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.195+0000 7f419b8a1700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4194106cd0 con 0x7f41940f5410 2024-02-19T22:10:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.195+0000 7f419963d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940f5410 0x7f41941067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.195+0000 7f419963d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940f5410 0x7f41941067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:50462/0 (socket says 172.21.15.184:50462) 2024-02-19T22:10:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.195+0000 7f419963d700 1 -- 172.21.15.184:0/1130755149 learned_addr learned my addr 172.21.15.184:0/1130755149 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.196+0000 7f419963d700 1 -- 172.21.15.184:0/1130755149 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4184005040 con 0x7f41940f5410 2024-02-19T22:10:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.196+0000 7f419963d700 1 --2- 172.21.15.184:0/1130755149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940f5410 0x7f41941067e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f4184005010 tx=0x7f418400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.196+0000 7f4189ffb700 1 -- 172.21.15.184:0/1130755149 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4184009320 con 0x7f41940f5410 2024-02-19T22:10:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.196+0000 7f419b8a1700 1 -- 172.21.15.184:0/1130755149 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4194106ed0 con 0x7f41940f5410 2024-02-19T22:10:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.196+0000 7f4189ffb700 1 -- 172.21.15.184:0/1130755149 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4184013380 con 0x7f41940f5410 2024-02-19T22:10:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.196+0000 7f4189ffb700 1 -- 172.21.15.184:0/1130755149 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4184014e50 con 0x7f41940f5410 2024-02-19T22:10:48.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.196+0000 7f419b8a1700 1 -- 172.21.15.184:0/1130755149 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4194107360 con 0x7f41940f5410 2024-02-19T22:10:48.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.197+0000 7f4189ffb700 1 -- 172.21.15.184:0/1130755149 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41840095f0 con 0x7f41940f5410 2024-02-19T22:10:48.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.197+0000 7f419b8a1700 1 -- 172.21.15.184:0/1130755149 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4194045ad0 con 0x7f41940f5410 2024-02-19T22:10:48.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.197+0000 7f4189ffb700 1 --2- 172.21.15.184:0/1130755149 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f418002ff10 0x7f41800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:48.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.197+0000 7f4189ffb700 1 -- 172.21.15.184:0/1130755149 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4184040fa0 con 0x7f41940f5410 2024-02-19T22:10:48.199 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.197+0000 7f4198e3c700 1 --2- 172.21.15.184:0/1130755149 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f418002ff10 0x7f41800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:48.200 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.198+0000 7f4198e3c700 1 --2- 172.21.15.184:0/1130755149 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f418002ff10 0x7f41800323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f41900096a0 tx=0x7f4190006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:48.203 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.202+0000 7f4189ffb700 1 -- 172.21.15.184:0/1130755149 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4184014420 con 0x7f41940f5410 2024-02-19T22:10:48.410 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.409+0000 7f419b8a1700 1 -- 172.21.15.184:0/1130755149 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4194108210 con 0x7f41940f5410 2024-02-19T22:10:48.411 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.410+0000 7f4189ffb700 1 -- 172.21.15.184:0/1130755149 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4184010230 con 0x7f41940f5410 2024-02-19T22:10:48.411 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:48.412 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:48.413 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.412+0000 7f419b8a1700 1 -- 172.21.15.184:0/1130755149 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f418002ff10 msgr2=0x7f41800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:48.413 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.412+0000 7f419b8a1700 1 --2- 172.21.15.184:0/1130755149 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f418002ff10 0x7f41800323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f41900096a0 tx=0x7f4190006b40).stop 2024-02-19T22:10:48.413 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.412+0000 7f419b8a1700 1 -- 172.21.15.184:0/1130755149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940f5410 msgr2=0x7f41941067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:48.413 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.412+0000 7f419b8a1700 1 --2- 172.21.15.184:0/1130755149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940f5410 0x7f41941067e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f4184005010 tx=0x7f418400b040).stop 2024-02-19T22:10:48.413 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.412+0000 7f419b8a1700 1 -- 172.21.15.184:0/1130755149 shutdown_connections 2024-02-19T22:10:48.413 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.412+0000 7f419b8a1700 1 --2- 172.21.15.184:0/1130755149 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f418002ff10 0x7f41800323d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:48.413 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.412+0000 7f419b8a1700 1 --2- 172.21.15.184:0/1130755149 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41940f5410 0x7f41941067e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:48.413 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.412+0000 7f419b8a1700 1 -- 172.21.15.184:0/1130755149 >> 172.21.15.184:0/1130755149 conn(0x7f41940f0b70 msgr2=0x7f41940f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:48.413 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.412+0000 7f419b8a1700 1 -- 172.21.15.184:0/1130755149 shutdown_connections 2024-02-19T22:10:48.413 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:48.412+0000 7f419b8a1700 1 -- 172.21.15.184:0/1130755149 wait complete. 2024-02-19T22:10:48.414 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:49.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:48 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:48.411224+0000 mon.smithi176 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.184:0/1130755149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:49.982 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:49.983 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:50.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:49 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:47.848436+0000 mgr.smithi176.faoogw (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:51.709 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.706+0000 7fb1eb59e700 1 -- 172.21.15.184:0/3931369125 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0f51a0 msgr2=0x7fb1ec0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:51.709 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.706+0000 7fb1eb59e700 1 --2- 172.21.15.184:0/3931369125 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0f51a0 0x7fb1ec0f55c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb1e0009fb0 tx=0x7fb1e0009b30).stop 2024-02-19T22:10:51.709 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.707+0000 7fb1eb59e700 1 -- 172.21.15.184:0/3931369125 shutdown_connections 2024-02-19T22:10:51.709 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.707+0000 7fb1eb59e700 1 --2- 172.21.15.184:0/3931369125 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0f51a0 0x7fb1ec0f55c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:51.709 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.707+0000 7fb1eb59e700 1 -- 172.21.15.184:0/3931369125 >> 172.21.15.184:0/3931369125 conn(0x7fb1ec0f09a0 msgr2=0x7fb1ec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:51.709 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.707+0000 7fb1eb59e700 1 -- 172.21.15.184:0/3931369125 shutdown_connections 2024-02-19T22:10:51.709 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.707+0000 7fb1eb59e700 1 -- 172.21.15.184:0/3931369125 wait complete. 2024-02-19T22:10:51.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.707+0000 7fb1eb59e700 1 Processor -- start 2024-02-19T22:10:51.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.707+0000 7fb1eb59e700 1 -- start start 2024-02-19T22:10:51.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.707+0000 7fb1eb59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0f51a0 0x7fb1ec1065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:51.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.707+0000 7fb1eb59e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec106ab0 con 0x7fb1ec0f51a0 2024-02-19T22:10:51.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.707+0000 7fb1ea59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0f51a0 0x7fb1ec1065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:51.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.708+0000 7fb1ea59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0f51a0 0x7fb1ec1065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:50474/0 (socket says 172.21.15.184:50474) 2024-02-19T22:10:51.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.708+0000 7fb1ea59c700 1 -- 172.21.15.184:0/806262560 learned_addr learned my addr 172.21.15.184:0/806262560 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:51.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.708+0000 7fb1ea59c700 1 -- 172.21.15.184:0/806262560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e0005040 con 0x7fb1ec0f51a0 2024-02-19T22:10:51.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.708+0000 7fb1ea59c700 1 --2- 172.21.15.184:0/806262560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0f51a0 0x7fb1ec1065c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb1e00089f0 tx=0x7fb1e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:51.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.708+0000 7fb1daffd700 1 -- 172.21.15.184:0/806262560 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e0009320 con 0x7fb1ec0f51a0 2024-02-19T22:10:51.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.708+0000 7fb1daffd700 1 -- 172.21.15.184:0/806262560 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1e0013380 con 0x7fb1ec0f51a0 2024-02-19T22:10:51.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.708+0000 7fb1eb59e700 1 -- 172.21.15.184:0/806262560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ec106cb0 con 0x7fb1ec0f51a0 2024-02-19T22:10:51.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.708+0000 7fb1daffd700 1 -- 172.21.15.184:0/806262560 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e0014e20 con 0x7fb1ec0f51a0 2024-02-19T22:10:51.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.708+0000 7fb1eb59e700 1 -- 172.21.15.184:0/806262560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ec107140 con 0x7fb1ec0f51a0 2024-02-19T22:10:51.711 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.709+0000 7fb1daffd700 1 -- 172.21.15.184:0/806262560 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1e00095f0 con 0x7fb1ec0f51a0 2024-02-19T22:10:51.711 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.709+0000 7fb1eb59e700 1 -- 172.21.15.184:0/806262560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ec045ad0 con 0x7fb1ec0f51a0 2024-02-19T22:10:51.712 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.709+0000 7fb1daffd700 1 --2- 172.21.15.184:0/806262560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb1d402ff10 0x7fb1d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:51.712 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.709+0000 7fb1daffd700 1 -- 172.21.15.184:0/806262560 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1e0040ed0 con 0x7fb1ec0f51a0 2024-02-19T22:10:51.712 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.709+0000 7fb1e9d9b700 1 --2- 172.21.15.184:0/806262560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb1d402ff10 0x7fb1d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:51.712 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.710+0000 7fb1e9d9b700 1 --2- 172.21.15.184:0/806262560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb1d402ff10 0x7fb1d40323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb1dc0096a0 tx=0x7fb1dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:51.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.714+0000 7fb1daffd700 1 -- 172.21.15.184:0/806262560 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1e0014420 con 0x7fb1ec0f51a0 2024-02-19T22:10:51.921 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.920+0000 7fb1eb59e700 1 -- 172.21.15.184:0/806262560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1ec107ff0 con 0x7fb1ec0f51a0 2024-02-19T22:10:51.921 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.921+0000 7fb1daffd700 1 -- 172.21.15.184:0/806262560 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb1e001d980 con 0x7fb1ec0f51a0 2024-02-19T22:10:51.922 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:51.922 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:51.923 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.922+0000 7fb1eb59e700 1 -- 172.21.15.184:0/806262560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb1d402ff10 msgr2=0x7fb1d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:51.923 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.922+0000 7fb1eb59e700 1 --2- 172.21.15.184:0/806262560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb1d402ff10 0x7fb1d40323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb1dc0096a0 tx=0x7fb1dc006b40).stop 2024-02-19T22:10:51.923 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.922+0000 7fb1eb59e700 1 -- 172.21.15.184:0/806262560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0f51a0 msgr2=0x7fb1ec1065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:51.924 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.922+0000 7fb1eb59e700 1 --2- 172.21.15.184:0/806262560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0f51a0 0x7fb1ec1065c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb1e00089f0 tx=0x7fb1e000b040).stop 2024-02-19T22:10:51.924 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.922+0000 7fb1eb59e700 1 -- 172.21.15.184:0/806262560 shutdown_connections 2024-02-19T22:10:51.924 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.922+0000 7fb1eb59e700 1 --2- 172.21.15.184:0/806262560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb1d402ff10 0x7fb1d40323d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:51.924 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.922+0000 7fb1eb59e700 1 --2- 172.21.15.184:0/806262560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb1ec0f51a0 0x7fb1ec1065c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:51.924 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.922+0000 7fb1eb59e700 1 -- 172.21.15.184:0/806262560 >> 172.21.15.184:0/806262560 conn(0x7fb1ec0f09a0 msgr2=0x7fb1ec05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:51.924 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.922+0000 7fb1eb59e700 1 -- 172.21.15.184:0/806262560 shutdown_connections 2024-02-19T22:10:51.924 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:51.922+0000 7fb1eb59e700 1 -- 172.21.15.184:0/806262560 wait complete. 2024-02-19T22:10:51.924 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:52.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:51 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:49.848788+0000 mgr.smithi176.faoogw (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:53.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:52 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:51.921953+0000 mon.smithi176 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.184:0/806262560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:53.488 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:53.489 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:54.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:51.849072+0000 mgr.smithi176.faoogw (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:55.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.252+0000 7f45fc945700 1 -- 172.21.15.184:0/3039111881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f80f6510 msgr2=0x7f45f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:55.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.252+0000 7f45fc945700 1 --2- 172.21.15.184:0/3039111881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f80f6510 0x7f45f80f6930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f45e8009fb0 tx=0x7f45e8009b30).stop 2024-02-19T22:10:55.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.252+0000 7f45fc945700 1 -- 172.21.15.184:0/3039111881 shutdown_connections 2024-02-19T22:10:55.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.252+0000 7f45fc945700 1 --2- 172.21.15.184:0/3039111881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f80f6510 0x7f45f80f6930 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:55.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.252+0000 7f45fc945700 1 -- 172.21.15.184:0/3039111881 >> 172.21.15.184:0/3039111881 conn(0x7f45f80f1c70 msgr2=0x7f45f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:55.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.252+0000 7f45fc945700 1 -- 172.21.15.184:0/3039111881 shutdown_connections 2024-02-19T22:10:55.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.252+0000 7f45fc945700 1 -- 172.21.15.184:0/3039111881 wait complete. 2024-02-19T22:10:55.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.252+0000 7f45fc945700 1 Processor -- start 2024-02-19T22:10:55.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.253+0000 7f45fc945700 1 -- start start 2024-02-19T22:10:55.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.253+0000 7f45fc945700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f80f6510 0x7f45f81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:55.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.253+0000 7f45fc945700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f8106cd0 con 0x7f45f80f6510 2024-02-19T22:10:55.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.253+0000 7f45f659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f80f6510 0x7f45f81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:55.276 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.253+0000 7f45f659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f80f6510 0x7f45f81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:50498/0 (socket says 172.21.15.184:50498) 2024-02-19T22:10:55.276 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.253+0000 7f45f659c700 1 -- 172.21.15.184:0/3689591532 learned_addr learned my addr 172.21.15.184:0/3689591532 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:55.276 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.253+0000 7f45f659c700 1 -- 172.21.15.184:0/3689591532 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e8005040 con 0x7f45f80f6510 2024-02-19T22:10:55.276 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.253+0000 7f45f659c700 1 --2- 172.21.15.184:0/3689591532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f80f6510 0x7f45f81067e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f45e8005010 tx=0x7f45e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:55.276 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.253+0000 7f45e6ffd700 1 -- 172.21.15.184:0/3689591532 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45e8009320 con 0x7f45f80f6510 2024-02-19T22:10:55.276 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.254+0000 7f45e6ffd700 1 -- 172.21.15.184:0/3689591532 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45e8013380 con 0x7f45f80f6510 2024-02-19T22:10:55.276 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.254+0000 7f45fc945700 1 -- 172.21.15.184:0/3689591532 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f8106ed0 con 0x7f45f80f6510 2024-02-19T22:10:55.276 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.254+0000 7f45e6ffd700 1 -- 172.21.15.184:0/3689591532 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45e8014e50 con 0x7f45f80f6510 2024-02-19T22:10:55.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.254+0000 7f45fc945700 1 -- 172.21.15.184:0/3689591532 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f8107360 con 0x7f45f80f6510 2024-02-19T22:10:55.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.254+0000 7f45e6ffd700 1 -- 172.21.15.184:0/3689591532 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45e80095f0 con 0x7f45f80f6510 2024-02-19T22:10:55.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.255+0000 7f45fc945700 1 -- 172.21.15.184:0/3689591532 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f8045ad0 con 0x7f45f80f6510 2024-02-19T22:10:55.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.255+0000 7f45e6ffd700 1 --2- 172.21.15.184:0/3689591532 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f45e002ff10 0x7f45e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:55.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.255+0000 7f45e6ffd700 1 -- 172.21.15.184:0/3689591532 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45e8040e90 con 0x7f45f80f6510 2024-02-19T22:10:55.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.255+0000 7f45f5d9b700 1 --2- 172.21.15.184:0/3689591532 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f45e002ff10 0x7f45e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:55.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.255+0000 7f45f5d9b700 1 --2- 172.21.15.184:0/3689591532 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f45e002ff10 0x7f45e00323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f45ec0096a0 tx=0x7f45ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:55.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.259+0000 7f45e6ffd700 1 -- 172.21.15.184:0/3689591532 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45e8014420 con 0x7f45f80f6510 2024-02-19T22:10:55.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.467+0000 7f45fc945700 1 -- 172.21.15.184:0/3689591532 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45f8108210 con 0x7f45f80f6510 2024-02-19T22:10:55.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.468+0000 7f45e6ffd700 1 -- 172.21.15.184:0/3689591532 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f45e8010230 con 0x7f45f80f6510 2024-02-19T22:10:55.469 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:55.469 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:55.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.469+0000 7f45fc945700 1 -- 172.21.15.184:0/3689591532 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f45e002ff10 msgr2=0x7f45e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:55.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.469+0000 7f45fc945700 1 --2- 172.21.15.184:0/3689591532 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f45e002ff10 0x7f45e00323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f45ec0096a0 tx=0x7f45ec006b40).stop 2024-02-19T22:10:55.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.469+0000 7f45fc945700 1 -- 172.21.15.184:0/3689591532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f80f6510 msgr2=0x7f45f81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:55.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.469+0000 7f45fc945700 1 --2- 172.21.15.184:0/3689591532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f80f6510 0x7f45f81067e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f45e8005010 tx=0x7f45e800b040).stop 2024-02-19T22:10:55.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.469+0000 7f45fc945700 1 -- 172.21.15.184:0/3689591532 shutdown_connections 2024-02-19T22:10:55.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.469+0000 7f45fc945700 1 --2- 172.21.15.184:0/3689591532 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f45e002ff10 0x7f45e00323d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:55.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.469+0000 7f45fc945700 1 --2- 172.21.15.184:0/3689591532 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45f80f6510 0x7f45f81067e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:55.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.469+0000 7f45fc945700 1 -- 172.21.15.184:0/3689591532 >> 172.21.15.184:0/3689591532 conn(0x7f45f80f1c70 msgr2=0x7f45f80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:55.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.469+0000 7f45fc945700 1 -- 172.21.15.184:0/3689591532 shutdown_connections 2024-02-19T22:10:55.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:55.469+0000 7f45fc945700 1 -- 172.21.15.184:0/3689591532 wait complete. 2024-02-19T22:10:55.471 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:10:56.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:55 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:53.849416+0000 mgr.smithi176.faoogw (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:56.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:55 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:55.468958+0000 mon.smithi176 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.184:0/3689591532' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:10:57.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:10:57.071 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:10:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:55.849741+0000 mgr.smithi176.faoogw (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:10:58.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.869+0000 7f38ed79f700 1 -- 172.21.15.184:0/3879833427 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38e80f51a0 msgr2=0x7f38e80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:58.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.869+0000 7f38ed79f700 1 --2- 172.21.15.184:0/3879833427 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38e80f51a0 0x7f38e80f55c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f38d8009fb0 tx=0x7f38d8009b30).stop 2024-02-19T22:10:58.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.869+0000 7f38ed79f700 1 -- 172.21.15.184:0/3879833427 shutdown_connections 2024-02-19T22:10:58.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.869+0000 7f38ed79f700 1 --2- 172.21.15.184:0/3879833427 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38e80f51a0 0x7f38e80f55c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:58.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.869+0000 7f38ed79f700 1 -- 172.21.15.184:0/3879833427 >> 172.21.15.184:0/3879833427 conn(0x7f38e80f09a0 msgr2=0x7f38e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:58.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.869+0000 7f38ed79f700 1 -- 172.21.15.184:0/3879833427 shutdown_connections 2024-02-19T22:10:58.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.869+0000 7f38ed79f700 1 -- 172.21.15.184:0/3879833427 wait complete. 2024-02-19T22:10:58.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.869+0000 7f38ed79f700 1 Processor -- start 2024-02-19T22:10:58.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.869+0000 7f38ed79f700 1 -- start start 2024-02-19T22:10:58.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.870+0000 7f38ed79f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38e80f51a0 0x7f38e8109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:58.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.870+0000 7f38ed79f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e8060210 con 0x7f38e80f51a0 2024-02-19T22:10:58.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.870+0000 7f38e7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38e80f51a0 0x7f38e8109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:58.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.870+0000 7f38e7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38e80f51a0 0x7f38e8109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:52010/0 (socket says 172.21.15.184:52010) 2024-02-19T22:10:58.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.870+0000 7f38e7fff700 1 -- 172.21.15.184:0/724378665 learned_addr learned my addr 172.21.15.184:0/724378665 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:10:58.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.870+0000 7f38e7fff700 1 -- 172.21.15.184:0/724378665 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d8005040 con 0x7f38e80f51a0 2024-02-19T22:10:58.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.870+0000 7f38e7fff700 1 --2- 172.21.15.184:0/724378665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38e80f51a0 0x7f38e8109290 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f38d80089f0 tx=0x7f38d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:58.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.870+0000 7f38e4ff9700 1 -- 172.21.15.184:0/724378665 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38d8009320 con 0x7f38e80f51a0 2024-02-19T22:10:58.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.870+0000 7f38ed79f700 1 -- 172.21.15.184:0/724378665 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e8060410 con 0x7f38e80f51a0 2024-02-19T22:10:58.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.870+0000 7f38e4ff9700 1 -- 172.21.15.184:0/724378665 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38d8013380 con 0x7f38e80f51a0 2024-02-19T22:10:58.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.870+0000 7f38e4ff9700 1 -- 172.21.15.184:0/724378665 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38d8014e20 con 0x7f38e80f51a0 2024-02-19T22:10:58.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.870+0000 7f38ed79f700 1 -- 172.21.15.184:0/724378665 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e8109940 con 0x7f38e80f51a0 2024-02-19T22:10:58.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.871+0000 7f38e4ff9700 1 -- 172.21.15.184:0/724378665 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38d80263a0 con 0x7f38e80f51a0 2024-02-19T22:10:58.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.871+0000 7f38ed79f700 1 -- 172.21.15.184:0/724378665 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e8059100 con 0x7f38e80f51a0 2024-02-19T22:10:58.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.871+0000 7f38e4ff9700 1 --2- 172.21.15.184:0/724378665 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f38d002ff60 0x7f38d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:10:58.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.871+0000 7f38e4ff9700 1 -- 172.21.15.184:0/724378665 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38d8041260 con 0x7f38e80f51a0 2024-02-19T22:10:58.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.872+0000 7f38e77fe700 1 --2- 172.21.15.184:0/724378665 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f38d002ff60 0x7f38d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:10:58.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.872+0000 7f38e77fe700 1 --2- 172.21.15.184:0/724378665 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f38d002ff60 0x7f38d0032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f38dc0096a0 tx=0x7f38dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:10:58.877 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:58.876+0000 7f38e4ff9700 1 -- 172.21.15.184:0/724378665 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38d80126f0 con 0x7f38e80f51a0 2024-02-19T22:10:59.082 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:59.081+0000 7f38ed79f700 1 -- 172.21.15.184:0/724378665 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38e80fad30 con 0x7f38e80f51a0 2024-02-19T22:10:59.082 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:59.082+0000 7f38e4ff9700 1 -- 172.21.15.184:0/724378665 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f38d80126f0 con 0x7f38e80f51a0 2024-02-19T22:10:59.083 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:10:59.083 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:10:59.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:59.083+0000 7f38ed79f700 1 -- 172.21.15.184:0/724378665 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f38d002ff60 msgr2=0x7f38d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:59.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:59.083+0000 7f38ed79f700 1 --2- 172.21.15.184:0/724378665 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f38d002ff60 0x7f38d0032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f38dc0096a0 tx=0x7f38dc006b40).stop 2024-02-19T22:10:59.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:59.083+0000 7f38ed79f700 1 -- 172.21.15.184:0/724378665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38e80f51a0 msgr2=0x7f38e8109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:10:59.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:59.083+0000 7f38ed79f700 1 --2- 172.21.15.184:0/724378665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38e80f51a0 0x7f38e8109290 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f38d80089f0 tx=0x7f38d800b040).stop 2024-02-19T22:10:59.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:59.083+0000 7f38ed79f700 1 -- 172.21.15.184:0/724378665 shutdown_connections 2024-02-19T22:10:59.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:59.083+0000 7f38ed79f700 1 --2- 172.21.15.184:0/724378665 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f38d002ff60 0x7f38d0032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:59.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:59.083+0000 7f38ed79f700 1 --2- 172.21.15.184:0/724378665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f38e80f51a0 0x7f38e8109290 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:10:59.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:59.083+0000 7f38ed79f700 1 -- 172.21.15.184:0/724378665 >> 172.21.15.184:0/724378665 conn(0x7f38e80f09a0 msgr2=0x7f38e80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:10:59.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:59.083+0000 7f38ed79f700 1 -- 172.21.15.184:0/724378665 shutdown_connections 2024-02-19T22:10:59.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:10:59.083+0000 7f38ed79f700 1 -- 172.21.15.184:0/724378665 wait complete. 2024-02-19T22:10:59.085 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:00.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:59 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:57.850114+0000 mgr.smithi176.faoogw (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:00.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:10:59 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:10:59.082948+0000 mon.smithi176 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.184:0/724378665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:00.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:00.695 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:02.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:10:59.850473+0000 mgr.smithi176.faoogw (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:02.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.571+0000 7fcca53cc700 1 -- 172.21.15.184:0/1180632657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcca00601e0 msgr2=0x7fcca00f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:02.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.571+0000 7fcca53cc700 1 --2- 172.21.15.184:0/1180632657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcca00601e0 0x7fcca00f9190 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fcc94009fb0 tx=0x7fcc94009b30).stop 2024-02-19T22:11:02.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.571+0000 7fcca53cc700 1 -- 172.21.15.184:0/1180632657 shutdown_connections 2024-02-19T22:11:02.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.571+0000 7fcca53cc700 1 --2- 172.21.15.184:0/1180632657 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcca00601e0 0x7fcca00f9190 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:02.573 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.571+0000 7fcca53cc700 1 -- 172.21.15.184:0/1180632657 >> 172.21.15.184:0/1180632657 conn(0x7fcca00f0b70 msgr2=0x7fcca00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:02.573 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.571+0000 7fcca53cc700 1 -- 172.21.15.184:0/1180632657 shutdown_connections 2024-02-19T22:11:02.573 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.571+0000 7fcca53cc700 1 -- 172.21.15.184:0/1180632657 wait complete. 2024-02-19T22:11:02.574 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.572+0000 7fcca53cc700 1 Processor -- start 2024-02-19T22:11:02.574 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.572+0000 7fcca53cc700 1 -- start start 2024-02-19T22:11:02.574 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.572+0000 7fcca53cc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcca00601e0 0x7fcca01067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:02.574 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.572+0000 7fcca53cc700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca0106cc0 con 0x7fcca00601e0 2024-02-19T22:11:02.574 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.572+0000 7fcc9effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcca00601e0 0x7fcca01067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:02.574 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.572+0000 7fcc9effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcca00601e0 0x7fcca01067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:52028/0 (socket says 172.21.15.184:52028) 2024-02-19T22:11:02.574 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.572+0000 7fcc9effd700 1 -- 172.21.15.184:0/1830479239 learned_addr learned my addr 172.21.15.184:0/1830479239 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:02.575 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.573+0000 7fcc9effd700 1 -- 172.21.15.184:0/1830479239 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc94005040 con 0x7fcca00601e0 2024-02-19T22:11:02.575 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.573+0000 7fcc9effd700 1 --2- 172.21.15.184:0/1830479239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcca00601e0 0x7fcca01067d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fcc94007a10 tx=0x7fcc94009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:02.575 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.573+0000 7fcc877fe700 1 -- 172.21.15.184:0/1830479239 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc94018070 con 0x7fcca00601e0 2024-02-19T22:11:02.575 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.573+0000 7fcca53cc700 1 -- 172.21.15.184:0/1830479239 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca0106ec0 con 0x7fcca00601e0 2024-02-19T22:11:02.575 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.573+0000 7fcc877fe700 1 -- 172.21.15.184:0/1830479239 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc94013410 con 0x7fcca00601e0 2024-02-19T22:11:02.575 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.573+0000 7fcc877fe700 1 -- 172.21.15.184:0/1830479239 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc9401c3c0 con 0x7fcca00601e0 2024-02-19T22:11:02.575 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.573+0000 7fcca53cc700 1 -- 172.21.15.184:0/1830479239 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca0107350 con 0x7fcca00601e0 2024-02-19T22:11:02.576 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.574+0000 7fcc877fe700 1 -- 172.21.15.184:0/1830479239 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc9401d420 con 0x7fcca00601e0 2024-02-19T22:11:02.576 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.574+0000 7fcca53cc700 1 -- 172.21.15.184:0/1830479239 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca0045ad0 con 0x7fcca00601e0 2024-02-19T22:11:02.576 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.574+0000 7fcc877fe700 1 --2- 172.21.15.184:0/1830479239 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcc8802ff60 0x7fcc88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:02.576 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.574+0000 7fcc877fe700 1 -- 172.21.15.184:0/1830479239 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc940414b0 con 0x7fcca00601e0 2024-02-19T22:11:02.576 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.574+0000 7fcc9e7fc700 1 --2- 172.21.15.184:0/1830479239 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcc8802ff60 0x7fcc88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:02.576 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.574+0000 7fcc9e7fc700 1 --2- 172.21.15.184:0/1830479239 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcc8802ff60 0x7fcc88032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fcc900096a0 tx=0x7fcc90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:02.580 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.579+0000 7fcc877fe700 1 -- 172.21.15.184:0/1830479239 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc9401d820 con 0x7fcca00601e0 2024-02-19T22:11:02.783 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.782+0000 7fcca53cc700 1 -- 172.21.15.184:0/1830479239 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcca0108200 con 0x7fcca00601e0 2024-02-19T22:11:02.783 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.782+0000 7fcc877fe700 1 -- 172.21.15.184:0/1830479239 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcc9401b370 con 0x7fcca00601e0 2024-02-19T22:11:02.783 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:02.783 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:02.785 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.784+0000 7fcca53cc700 1 -- 172.21.15.184:0/1830479239 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcc8802ff60 msgr2=0x7fcc88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:02.785 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.784+0000 7fcca53cc700 1 --2- 172.21.15.184:0/1830479239 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcc8802ff60 0x7fcc88032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fcc900096a0 tx=0x7fcc90006b40).stop 2024-02-19T22:11:02.785 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.784+0000 7fcca53cc700 1 -- 172.21.15.184:0/1830479239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcca00601e0 msgr2=0x7fcca01067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:02.785 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.784+0000 7fcca53cc700 1 --2- 172.21.15.184:0/1830479239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcca00601e0 0x7fcca01067d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fcc94007a10 tx=0x7fcc94009b30).stop 2024-02-19T22:11:02.785 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.784+0000 7fcca53cc700 1 -- 172.21.15.184:0/1830479239 shutdown_connections 2024-02-19T22:11:02.785 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.784+0000 7fcca53cc700 1 --2- 172.21.15.184:0/1830479239 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcc8802ff60 0x7fcc88032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:02.786 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.784+0000 7fcca53cc700 1 --2- 172.21.15.184:0/1830479239 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcca00601e0 0x7fcca01067d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:02.786 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.784+0000 7fcca53cc700 1 -- 172.21.15.184:0/1830479239 >> 172.21.15.184:0/1830479239 conn(0x7fcca00f0b70 msgr2=0x7fcca00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:02.786 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.784+0000 7fcca53cc700 1 -- 172.21.15.184:0/1830479239 shutdown_connections 2024-02-19T22:11:02.786 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:02.784+0000 7fcca53cc700 1 -- 172.21.15.184:0/1830479239 wait complete. 2024-02-19T22:11:02.786 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:04.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:01.850765+0000 mgr.smithi176.faoogw (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:04.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:02.783661+0000 mon.smithi176 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.184:0/1830479239' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:04.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:04.448 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:06.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:03.851083+0000 mgr.smithi176.faoogw (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:06.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.170+0000 7fc8d5880700 1 -- 172.21.15.184:0/1151591930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d00f51a0 msgr2=0x7fc8d00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:06.187 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.170+0000 7fc8d5880700 1 --2- 172.21.15.184:0/1151591930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d00f51a0 0x7fc8d00f55c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fc8c4009fb0 tx=0x7fc8c4009b30).stop 2024-02-19T22:11:06.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.170+0000 7fc8d5880700 1 -- 172.21.15.184:0/1151591930 shutdown_connections 2024-02-19T22:11:06.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.170+0000 7fc8d5880700 1 --2- 172.21.15.184:0/1151591930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d00f51a0 0x7fc8d00f55c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:06.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.170+0000 7fc8d5880700 1 -- 172.21.15.184:0/1151591930 >> 172.21.15.184:0/1151591930 conn(0x7fc8d00f09a0 msgr2=0x7fc8d00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:06.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.170+0000 7fc8d5880700 1 -- 172.21.15.184:0/1151591930 shutdown_connections 2024-02-19T22:11:06.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.170+0000 7fc8d5880700 1 -- 172.21.15.184:0/1151591930 wait complete. 2024-02-19T22:11:06.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.170+0000 7fc8d5880700 1 Processor -- start 2024-02-19T22:11:06.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.170+0000 7fc8d5880700 1 -- start start 2024-02-19T22:11:06.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.171+0000 7fc8d5880700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d00f51a0 0x7fc8d01054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:06.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.171+0000 7fc8d5880700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d01059c0 con 0x7fc8d00f51a0 2024-02-19T22:11:06.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.171+0000 7fc8d487e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d00f51a0 0x7fc8d01054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:06.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.171+0000 7fc8d487e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d00f51a0 0x7fc8d01054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:52042/0 (socket says 172.21.15.184:52042) 2024-02-19T22:11:06.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.171+0000 7fc8d487e700 1 -- 172.21.15.184:0/2957821623 learned_addr learned my addr 172.21.15.184:0/2957821623 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:06.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.171+0000 7fc8d487e700 1 -- 172.21.15.184:0/2957821623 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c4005040 con 0x7fc8d00f51a0 2024-02-19T22:11:06.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.171+0000 7fc8d487e700 1 --2- 172.21.15.184:0/2957821623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d00f51a0 0x7fc8d01054d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fc8c40089f0 tx=0x7fc8c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:06.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.171+0000 7fc8cd7fa700 1 -- 172.21.15.184:0/2957821623 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8c4009320 con 0x7fc8d00f51a0 2024-02-19T22:11:06.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.171+0000 7fc8d5880700 1 -- 172.21.15.184:0/2957821623 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d0105bc0 con 0x7fc8d00f51a0 2024-02-19T22:11:06.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.171+0000 7fc8cd7fa700 1 -- 172.21.15.184:0/2957821623 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8c4013380 con 0x7fc8d00f51a0 2024-02-19T22:11:06.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.171+0000 7fc8cd7fa700 1 -- 172.21.15.184:0/2957821623 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8c4014e20 con 0x7fc8d00f51a0 2024-02-19T22:11:06.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.171+0000 7fc8d5880700 1 -- 172.21.15.184:0/2957821623 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d0106050 con 0x7fc8d00f51a0 2024-02-19T22:11:06.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.172+0000 7fc8cd7fa700 1 -- 172.21.15.184:0/2957821623 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8c40095f0 con 0x7fc8d00f51a0 2024-02-19T22:11:06.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.172+0000 7fc8d5880700 1 -- 172.21.15.184:0/2957821623 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d0045ad0 con 0x7fc8d00f51a0 2024-02-19T22:11:06.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.172+0000 7fc8cd7fa700 1 --2- 172.21.15.184:0/2957821623 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc8c002ff10 0x7fc8c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:06.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.172+0000 7fc8cd7fa700 1 -- 172.21.15.184:0/2957821623 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc8c4040e90 con 0x7fc8d00f51a0 2024-02-19T22:11:06.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.173+0000 7fc8cffff700 1 --2- 172.21.15.184:0/2957821623 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc8c002ff10 0x7fc8c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:06.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.173+0000 7fc8cffff700 1 --2- 172.21.15.184:0/2957821623 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc8c002ff10 0x7fc8c00323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fc8bc0096a0 tx=0x7fc8bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:06.189 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.177+0000 7fc8cd7fa700 1 -- 172.21.15.184:0/2957821623 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc8c4014420 con 0x7fc8d00f51a0 2024-02-19T22:11:06.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.381+0000 7fc8d5880700 1 -- 172.21.15.184:0/2957821623 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8d0106f00 con 0x7fc8d00f51a0 2024-02-19T22:11:06.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.382+0000 7fc8cd7fa700 1 -- 172.21.15.184:0/2957821623 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc8c401d980 con 0x7fc8d00f51a0 2024-02-19T22:11:06.383 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:06.383 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:06.384 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.383+0000 7fc8d5880700 1 -- 172.21.15.184:0/2957821623 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc8c002ff10 msgr2=0x7fc8c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:06.384 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.383+0000 7fc8d5880700 1 --2- 172.21.15.184:0/2957821623 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc8c002ff10 0x7fc8c00323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fc8bc0096a0 tx=0x7fc8bc006b40).stop 2024-02-19T22:11:06.384 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.383+0000 7fc8d5880700 1 -- 172.21.15.184:0/2957821623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d00f51a0 msgr2=0x7fc8d01054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:06.384 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.383+0000 7fc8d5880700 1 --2- 172.21.15.184:0/2957821623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d00f51a0 0x7fc8d01054d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fc8c40089f0 tx=0x7fc8c400b040).stop 2024-02-19T22:11:06.385 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.383+0000 7fc8d5880700 1 -- 172.21.15.184:0/2957821623 shutdown_connections 2024-02-19T22:11:06.385 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.383+0000 7fc8d5880700 1 --2- 172.21.15.184:0/2957821623 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc8c002ff10 0x7fc8c00323d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:06.385 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.383+0000 7fc8d5880700 1 --2- 172.21.15.184:0/2957821623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8d00f51a0 0x7fc8d01054d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:06.385 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.383+0000 7fc8d5880700 1 -- 172.21.15.184:0/2957821623 >> 172.21.15.184:0/2957821623 conn(0x7fc8d00f09a0 msgr2=0x7fc8d00f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:06.385 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.383+0000 7fc8d5880700 1 -- 172.21.15.184:0/2957821623 shutdown_connections 2024-02-19T22:11:06.385 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:06.383+0000 7fc8d5880700 1 -- 172.21.15.184:0/2957821623 wait complete. 2024-02-19T22:11:06.385 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:07.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:06.383056+0000 mon.smithi176 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.184:0/2957821623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:08.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:08.022 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:08.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:05.851419+0000 mgr.smithi176.faoogw (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:09.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.883+0000 7f4286d66700 1 -- 172.21.15.184:0/3065524488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42800f5410 msgr2=0x7f42800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:09.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.883+0000 7f4286d66700 1 --2- 172.21.15.184:0/3065524488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42800f5410 0x7f42800f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4274009fb0 tx=0x7f4274009b30).stop 2024-02-19T22:11:09.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.883+0000 7f4286d66700 1 -- 172.21.15.184:0/3065524488 shutdown_connections 2024-02-19T22:11:09.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.883+0000 7f4286d66700 1 --2- 172.21.15.184:0/3065524488 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42800f5410 0x7f42800f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:09.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.883+0000 7f4286d66700 1 -- 172.21.15.184:0/3065524488 >> 172.21.15.184:0/3065524488 conn(0x7f42800f0b70 msgr2=0x7f42800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:09.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.883+0000 7f4286d66700 1 -- 172.21.15.184:0/3065524488 shutdown_connections 2024-02-19T22:11:09.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.883+0000 7f4286d66700 1 -- 172.21.15.184:0/3065524488 wait complete. 2024-02-19T22:11:09.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.884+0000 7f4286d66700 1 Processor -- start 2024-02-19T22:11:09.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.884+0000 7f4286d66700 1 -- start start 2024-02-19T22:11:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.884+0000 7f4286d66700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42800f5410 0x7f4280109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.884+0000 7f4286d66700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4280060220 con 0x7f42800f5410 2024-02-19T22:11:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.884+0000 7f4284b02700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42800f5410 0x7f4280109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.884+0000 7f4284b02700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42800f5410 0x7f4280109530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:43184/0 (socket says 172.21.15.184:43184) 2024-02-19T22:11:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.884+0000 7f4284b02700 1 -- 172.21.15.184:0/3070006089 learned_addr learned my addr 172.21.15.184:0/3070006089 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.885+0000 7f4284b02700 1 -- 172.21.15.184:0/3070006089 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4274005040 con 0x7f42800f5410 2024-02-19T22:11:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.885+0000 7f4284b02700 1 --2- 172.21.15.184:0/3070006089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42800f5410 0x7f4280109530 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4274005010 tx=0x7f427400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.885+0000 7f427d7fa700 1 -- 172.21.15.184:0/3070006089 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4274009320 con 0x7f42800f5410 2024-02-19T22:11:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.885+0000 7f4286d66700 1 -- 172.21.15.184:0/3070006089 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4280060420 con 0x7f42800f5410 2024-02-19T22:11:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.885+0000 7f427d7fa700 1 -- 172.21.15.184:0/3070006089 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4274013380 con 0x7f42800f5410 2024-02-19T22:11:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.885+0000 7f427d7fa700 1 -- 172.21.15.184:0/3070006089 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4274014e50 con 0x7f42800f5410 2024-02-19T22:11:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.885+0000 7f4286d66700 1 -- 172.21.15.184:0/3070006089 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4280109be0 con 0x7f42800f5410 2024-02-19T22:11:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.886+0000 7f427d7fa700 1 -- 172.21.15.184:0/3070006089 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4274014420 con 0x7f42800f5410 2024-02-19T22:11:09.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.886+0000 7f4286d66700 1 -- 172.21.15.184:0/3070006089 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4280059100 con 0x7f42800f5410 2024-02-19T22:11:09.935 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.886+0000 7f427d7fa700 1 --2- 172.21.15.184:0/3070006089 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f427002ff10 0x7f42700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:09.935 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.886+0000 7f427d7fa700 1 -- 172.21.15.184:0/3070006089 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4274040e30 con 0x7f42800f5410 2024-02-19T22:11:09.935 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.886+0000 7f427ffff700 1 --2- 172.21.15.184:0/3070006089 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f427002ff10 0x7f42700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:09.935 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.886+0000 7f427ffff700 1 --2- 172.21.15.184:0/3070006089 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f427002ff10 0x7f42700323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f426c0096a0 tx=0x7f426c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:09.935 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:09.891+0000 7f427d7fa700 1 -- 172.21.15.184:0/3070006089 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42740095f0 con 0x7f42800f5410 2024-02-19T22:11:10.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:10.096+0000 7f4286d66700 1 -- 172.21.15.184:0/3070006089 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4280045ad0 con 0x7f42800f5410 2024-02-19T22:11:10.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:10.097+0000 7f427d7fa700 1 -- 172.21.15.184:0/3070006089 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f427401db70 con 0x7f42800f5410 2024-02-19T22:11:10.098 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:10.098 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:10.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:10.098+0000 7f4286d66700 1 -- 172.21.15.184:0/3070006089 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f427002ff10 msgr2=0x7f42700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:10.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:10.098+0000 7f4286d66700 1 --2- 172.21.15.184:0/3070006089 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f427002ff10 0x7f42700323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f426c0096a0 tx=0x7f426c006b40).stop 2024-02-19T22:11:10.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:10.098+0000 7f4286d66700 1 -- 172.21.15.184:0/3070006089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42800f5410 msgr2=0x7f4280109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:10.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:10.098+0000 7f4286d66700 1 --2- 172.21.15.184:0/3070006089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42800f5410 0x7f4280109530 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4274005010 tx=0x7f427400b040).stop 2024-02-19T22:11:10.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:10.099+0000 7f4286d66700 1 -- 172.21.15.184:0/3070006089 shutdown_connections 2024-02-19T22:11:10.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:10.099+0000 7f4286d66700 1 --2- 172.21.15.184:0/3070006089 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f427002ff10 0x7f42700323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:10.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:10.099+0000 7f4286d66700 1 --2- 172.21.15.184:0/3070006089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42800f5410 0x7f4280109530 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:10.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:10.099+0000 7f4286d66700 1 -- 172.21.15.184:0/3070006089 >> 172.21.15.184:0/3070006089 conn(0x7f42800f0b70 msgr2=0x7f42800f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:10.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:10.099+0000 7f4286d66700 1 -- 172.21.15.184:0/3070006089 shutdown_connections 2024-02-19T22:11:10.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:10.099+0000 7f4286d66700 1 -- 172.21.15.184:0/3070006089 wait complete. 2024-02-19T22:11:10.101 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:10.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:07.851733+0000 mgr.smithi176.faoogw (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:11.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:10 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:10.098254+0000 mon.smithi176 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.184:0/3070006089' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:11.737 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:11.737 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:12.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:09.851992+0000 mgr.smithi176.faoogw (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:13.542 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.540+0000 7fb8666a8700 1 -- 172.21.15.184:0/1577769914 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8600f5410 msgr2=0x7fb8600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:13.542 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.540+0000 7fb8666a8700 1 --2- 172.21.15.184:0/1577769914 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8600f5410 0x7fb8600f5830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fb854009fb0 tx=0x7fb854009b30).stop 2024-02-19T22:11:13.542 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.540+0000 7fb8666a8700 1 -- 172.21.15.184:0/1577769914 shutdown_connections 2024-02-19T22:11:13.542 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.540+0000 7fb8666a8700 1 --2- 172.21.15.184:0/1577769914 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8600f5410 0x7fb8600f5830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.540+0000 7fb8666a8700 1 -- 172.21.15.184:0/1577769914 >> 172.21.15.184:0/1577769914 conn(0x7fb8600f0b70 msgr2=0x7fb8600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.540+0000 7fb8666a8700 1 -- 172.21.15.184:0/1577769914 shutdown_connections 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.540+0000 7fb8666a8700 1 -- 172.21.15.184:0/1577769914 wait complete. 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.541+0000 7fb8666a8700 1 Processor -- start 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.541+0000 7fb8666a8700 1 -- start start 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.541+0000 7fb8666a8700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8600f5410 0x7fb8601067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.541+0000 7fb8666a8700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb860106cd0 con 0x7fb8600f5410 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.541+0000 7fb85ffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8600f5410 0x7fb8601067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.541+0000 7fb85ffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8600f5410 0x7fb8601067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:43190/0 (socket says 172.21.15.184:43190) 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.541+0000 7fb85ffff700 1 -- 172.21.15.184:0/1897344836 learned_addr learned my addr 172.21.15.184:0/1897344836 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.542+0000 7fb85ffff700 1 -- 172.21.15.184:0/1897344836 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb854005040 con 0x7fb8600f5410 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.542+0000 7fb85ffff700 1 --2- 172.21.15.184:0/1897344836 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8600f5410 0x7fb8601067e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fb854005010 tx=0x7fb85400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.542+0000 7fb85cff9700 1 -- 172.21.15.184:0/1897344836 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb854009320 con 0x7fb8600f5410 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.542+0000 7fb8666a8700 1 -- 172.21.15.184:0/1897344836 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb860106ed0 con 0x7fb8600f5410 2024-02-19T22:11:13.543 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.542+0000 7fb85cff9700 1 -- 172.21.15.184:0/1897344836 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb854013380 con 0x7fb8600f5410 2024-02-19T22:11:13.544 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.542+0000 7fb85cff9700 1 -- 172.21.15.184:0/1897344836 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb854014e50 con 0x7fb8600f5410 2024-02-19T22:11:13.544 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.542+0000 7fb8666a8700 1 -- 172.21.15.184:0/1897344836 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb860107360 con 0x7fb8600f5410 2024-02-19T22:11:13.545 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.543+0000 7fb85cff9700 1 -- 172.21.15.184:0/1897344836 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8540263a0 con 0x7fb8600f5410 2024-02-19T22:11:13.545 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.543+0000 7fb85cff9700 1 --2- 172.21.15.184:0/1897344836 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb84802ff10 0x7fb8480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:13.545 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.543+0000 7fb85cff9700 1 -- 172.21.15.184:0/1897344836 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8540095f0 con 0x7fb8600f5410 2024-02-19T22:11:13.545 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.543+0000 7fb8666a8700 1 -- 172.21.15.184:0/1897344836 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb860045ad0 con 0x7fb8600f5410 2024-02-19T22:11:13.545 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.543+0000 7fb85f7fe700 1 --2- 172.21.15.184:0/1897344836 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb84802ff10 0x7fb8480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:13.545 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.543+0000 7fb85f7fe700 1 --2- 172.21.15.184:0/1897344836 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb84802ff10 0x7fb8480323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb8500096a0 tx=0x7fb850006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:13.548 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.548+0000 7fb85cff9700 1 -- 172.21.15.184:0/1897344836 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb854012920 con 0x7fb8600f5410 2024-02-19T22:11:13.753 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.753+0000 7fb8666a8700 1 -- 172.21.15.184:0/1897344836 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb860108080 con 0x7fb8600f5410 2024-02-19T22:11:13.753 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.753+0000 7fb85cff9700 1 -- 172.21.15.184:0/1897344836 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb85402b020 con 0x7fb8600f5410 2024-02-19T22:11:13.753 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:13.753 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:13.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.755+0000 7fb8666a8700 1 -- 172.21.15.184:0/1897344836 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb84802ff10 msgr2=0x7fb8480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:13.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.755+0000 7fb8666a8700 1 --2- 172.21.15.184:0/1897344836 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb84802ff10 0x7fb8480323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb8500096a0 tx=0x7fb850006b40).stop 2024-02-19T22:11:13.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.755+0000 7fb8666a8700 1 -- 172.21.15.184:0/1897344836 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8600f5410 msgr2=0x7fb8601067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:13.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.755+0000 7fb8666a8700 1 --2- 172.21.15.184:0/1897344836 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8600f5410 0x7fb8601067e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fb854005010 tx=0x7fb85400b040).stop 2024-02-19T22:11:13.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.755+0000 7fb8666a8700 1 -- 172.21.15.184:0/1897344836 shutdown_connections 2024-02-19T22:11:13.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.755+0000 7fb8666a8700 1 --2- 172.21.15.184:0/1897344836 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb84802ff10 0x7fb8480323d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:13.755 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.755+0000 7fb8666a8700 1 --2- 172.21.15.184:0/1897344836 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb8600f5410 0x7fb8601067e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:13.756 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.755+0000 7fb8666a8700 1 -- 172.21.15.184:0/1897344836 >> 172.21.15.184:0/1897344836 conn(0x7fb8600f0b70 msgr2=0x7fb86005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:13.756 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.755+0000 7fb8666a8700 1 -- 172.21.15.184:0/1897344836 shutdown_connections 2024-02-19T22:11:13.756 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:13.755+0000 7fb8666a8700 1 -- 172.21.15.184:0/1897344836 wait complete. 2024-02-19T22:11:13.756 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:14.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:13 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:11.852251+0000 mgr.smithi176.faoogw (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:14.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:13 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:13.753683+0000 mon.smithi176 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.184:0/1897344836' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:15.425 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:15.425 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:16.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:13.852618+0000 mgr.smithi176.faoogw (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:17.228 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.228+0000 7fc558f83700 1 -- 172.21.15.184:0/521944105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5540f5410 msgr2=0x7fc5540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:17.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.228+0000 7fc558f83700 1 --2- 172.21.15.184:0/521944105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5540f5410 0x7fc5540f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fc53c009fb0 tx=0x7fc53c009b30).stop 2024-02-19T22:11:17.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.228+0000 7fc558f83700 1 -- 172.21.15.184:0/521944105 shutdown_connections 2024-02-19T22:11:17.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.228+0000 7fc558f83700 1 --2- 172.21.15.184:0/521944105 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5540f5410 0x7fc5540f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:17.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.228+0000 7fc558f83700 1 -- 172.21.15.184:0/521944105 >> 172.21.15.184:0/521944105 conn(0x7fc5540f0b70 msgr2=0x7fc5540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:17.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.228+0000 7fc558f83700 1 -- 172.21.15.184:0/521944105 shutdown_connections 2024-02-19T22:11:17.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.228+0000 7fc558f83700 1 -- 172.21.15.184:0/521944105 wait complete. 2024-02-19T22:11:17.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.228+0000 7fc558f83700 1 Processor -- start 2024-02-19T22:11:17.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.228+0000 7fc558f83700 1 -- start start 2024-02-19T22:11:17.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.228+0000 7fc558f83700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5540f5410 0x7fc5541067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:17.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.228+0000 7fc558f83700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc554106cd0 con 0x7fc5540f5410 2024-02-19T22:11:17.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.229+0000 7fc55259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5540f5410 0x7fc5541067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:17.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.229+0000 7fc55259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5540f5410 0x7fc5541067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:34376/0 (socket says 172.21.15.184:34376) 2024-02-19T22:11:17.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.229+0000 7fc55259c700 1 -- 172.21.15.184:0/1725404475 learned_addr learned my addr 172.21.15.184:0/1725404475 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:17.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.229+0000 7fc55259c700 1 -- 172.21.15.184:0/1725404475 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c005040 con 0x7fc5540f5410 2024-02-19T22:11:17.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.229+0000 7fc55259c700 1 --2- 172.21.15.184:0/1725404475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5540f5410 0x7fc5541067e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc53c005010 tx=0x7fc53c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:17.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.229+0000 7fc54affd700 1 -- 172.21.15.184:0/1725404475 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc53c009320 con 0x7fc5540f5410 2024-02-19T22:11:17.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.229+0000 7fc558f83700 1 -- 172.21.15.184:0/1725404475 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc554106ed0 con 0x7fc5540f5410 2024-02-19T22:11:17.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.229+0000 7fc54affd700 1 -- 172.21.15.184:0/1725404475 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc53c013380 con 0x7fc5540f5410 2024-02-19T22:11:17.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.229+0000 7fc54affd700 1 -- 172.21.15.184:0/1725404475 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc53c014e50 con 0x7fc5540f5410 2024-02-19T22:11:17.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.229+0000 7fc558f83700 1 -- 172.21.15.184:0/1725404475 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc554107360 con 0x7fc5540f5410 2024-02-19T22:11:17.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.230+0000 7fc54affd700 1 -- 172.21.15.184:0/1725404475 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc53c014420 con 0x7fc5540f5410 2024-02-19T22:11:17.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.230+0000 7fc558f83700 1 -- 172.21.15.184:0/1725404475 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc554045ad0 con 0x7fc5540f5410 2024-02-19T22:11:17.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.230+0000 7fc54affd700 1 --2- 172.21.15.184:0/1725404475 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc54002ff10 0x7fc5400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:17.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.230+0000 7fc54affd700 1 -- 172.21.15.184:0/1725404475 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc53c040eb0 con 0x7fc5540f5410 2024-02-19T22:11:17.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.230+0000 7fc551d9b700 1 --2- 172.21.15.184:0/1725404475 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc54002ff10 0x7fc5400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:17.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.231+0000 7fc551d9b700 1 --2- 172.21.15.184:0/1725404475 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc54002ff10 0x7fc5400323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc5440096a0 tx=0x7fc544006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:17.235 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.235+0000 7fc54affd700 1 -- 172.21.15.184:0/1725404475 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc53c0095f0 con 0x7fc5540f5410 2024-02-19T22:11:17.440 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.439+0000 7fc558f83700 1 -- 172.21.15.184:0/1725404475 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc554108210 con 0x7fc5540f5410 2024-02-19T22:11:17.441 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.439+0000 7fc54affd700 1 -- 172.21.15.184:0/1725404475 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc53c01db70 con 0x7fc5540f5410 2024-02-19T22:11:17.441 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:17.441 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:17.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.441+0000 7fc558f83700 1 -- 172.21.15.184:0/1725404475 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc54002ff10 msgr2=0x7fc5400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:17.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.441+0000 7fc558f83700 1 --2- 172.21.15.184:0/1725404475 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc54002ff10 0x7fc5400323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fc5440096a0 tx=0x7fc544006b40).stop 2024-02-19T22:11:17.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.441+0000 7fc558f83700 1 -- 172.21.15.184:0/1725404475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5540f5410 msgr2=0x7fc5541067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:17.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.441+0000 7fc558f83700 1 --2- 172.21.15.184:0/1725404475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5540f5410 0x7fc5541067e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc53c005010 tx=0x7fc53c00b040).stop 2024-02-19T22:11:17.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.441+0000 7fc558f83700 1 -- 172.21.15.184:0/1725404475 shutdown_connections 2024-02-19T22:11:17.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.441+0000 7fc558f83700 1 --2- 172.21.15.184:0/1725404475 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc54002ff10 0x7fc5400323d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:17.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.441+0000 7fc558f83700 1 --2- 172.21.15.184:0/1725404475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc5540f5410 0x7fc5541067e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:17.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.441+0000 7fc558f83700 1 -- 172.21.15.184:0/1725404475 >> 172.21.15.184:0/1725404475 conn(0x7fc5540f0b70 msgr2=0x7fc5540f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:17.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.441+0000 7fc558f83700 1 -- 172.21.15.184:0/1725404475 shutdown_connections 2024-02-19T22:11:17.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:17.441+0000 7fc558f83700 1 -- 172.21.15.184:0/1725404475 wait complete. 2024-02-19T22:11:17.443 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:18.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:17 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:15.852867+0000 mgr.smithi176.faoogw (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:18.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:17 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:17.440729+0000 mon.smithi176 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.184:0/1725404475' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:19.115 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:19.115 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:20.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:17.853143+0000 mgr.smithi176.faoogw (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:21.034 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.032+0000 7f639ff88700 1 -- 172.21.15.184:0/982251447 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63980f5410 msgr2=0x7f63980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:21.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.032+0000 7f639ff88700 1 --2- 172.21.15.184:0/982251447 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63980f5410 0x7f63980f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f638c009fb0 tx=0x7f638c009b30).stop 2024-02-19T22:11:21.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.032+0000 7f639ff88700 1 -- 172.21.15.184:0/982251447 shutdown_connections 2024-02-19T22:11:21.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.032+0000 7f639ff88700 1 --2- 172.21.15.184:0/982251447 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63980f5410 0x7f63980f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:21.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.032+0000 7f639ff88700 1 -- 172.21.15.184:0/982251447 >> 172.21.15.184:0/982251447 conn(0x7f63980f0b70 msgr2=0x7f63980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:21.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.032+0000 7f639ff88700 1 -- 172.21.15.184:0/982251447 shutdown_connections 2024-02-19T22:11:21.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.032+0000 7f639ff88700 1 -- 172.21.15.184:0/982251447 wait complete. 2024-02-19T22:11:21.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.032+0000 7f639ff88700 1 Processor -- start 2024-02-19T22:11:21.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.032+0000 7f639ff88700 1 -- start start 2024-02-19T22:11:21.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.032+0000 7f639ff88700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63980f5410 0x7f6398108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:21.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.032+0000 7f639ff88700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6398108ef0 con 0x7f63980f5410 2024-02-19T22:11:21.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.033+0000 7f639dd24700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63980f5410 0x7f6398108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:21.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.033+0000 7f639dd24700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63980f5410 0x7f6398108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:34392/0 (socket says 172.21.15.184:34392) 2024-02-19T22:11:21.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.033+0000 7f639dd24700 1 -- 172.21.15.184:0/2596532816 learned_addr learned my addr 172.21.15.184:0/2596532816 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:21.036 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.033+0000 7f639dd24700 1 -- 172.21.15.184:0/2596532816 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c005040 con 0x7f63980f5410 2024-02-19T22:11:21.036 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.033+0000 7f639dd24700 1 --2- 172.21.15.184:0/2596532816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63980f5410 0x7f6398108a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f638c005010 tx=0x7f638c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:21.036 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.033+0000 7f638a7fc700 1 -- 172.21.15.184:0/2596532816 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f638c009320 con 0x7f63980f5410 2024-02-19T22:11:21.036 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.033+0000 7f639ff88700 1 -- 172.21.15.184:0/2596532816 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63981090f0 con 0x7f63980f5410 2024-02-19T22:11:21.036 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.033+0000 7f638a7fc700 1 -- 172.21.15.184:0/2596532816 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f638c013380 con 0x7f63980f5410 2024-02-19T22:11:21.036 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.033+0000 7f638a7fc700 1 -- 172.21.15.184:0/2596532816 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f638c014e50 con 0x7f63980f5410 2024-02-19T22:11:21.036 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.033+0000 7f639ff88700 1 -- 172.21.15.184:0/2596532816 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6398109580 con 0x7f63980f5410 2024-02-19T22:11:21.037 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.034+0000 7f638a7fc700 1 -- 172.21.15.184:0/2596532816 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f638c014420 con 0x7f63980f5410 2024-02-19T22:11:21.037 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.034+0000 7f639ff88700 1 -- 172.21.15.184:0/2596532816 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6398045ad0 con 0x7f63980f5410 2024-02-19T22:11:21.037 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.034+0000 7f638a7fc700 1 --2- 172.21.15.184:0/2596532816 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f638402ff10 0x7f63840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:21.037 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.034+0000 7f638a7fc700 1 -- 172.21.15.184:0/2596532816 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f638c040f70 con 0x7f63980f5410 2024-02-19T22:11:21.037 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.034+0000 7f639d523700 1 --2- 172.21.15.184:0/2596532816 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f638402ff10 0x7f63840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:21.037 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.035+0000 7f639d523700 1 --2- 172.21.15.184:0/2596532816 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f638402ff10 0x7f63840323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f63940096a0 tx=0x7f6394006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:21.040 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.039+0000 7f638a7fc700 1 -- 172.21.15.184:0/2596532816 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f638c0095f0 con 0x7f63980f5410 2024-02-19T22:11:21.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.246+0000 7f639ff88700 1 -- 172.21.15.184:0/2596532816 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f639810a430 con 0x7f63980f5410 2024-02-19T22:11:21.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.247+0000 7f638a7fc700 1 -- 172.21.15.184:0/2596532816 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f638c01db70 con 0x7f63980f5410 2024-02-19T22:11:21.248 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:21.248 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:21.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.248+0000 7f639ff88700 1 -- 172.21.15.184:0/2596532816 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f638402ff10 msgr2=0x7f63840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:21.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.248+0000 7f639ff88700 1 --2- 172.21.15.184:0/2596532816 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f638402ff10 0x7f63840323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f63940096a0 tx=0x7f6394006b40).stop 2024-02-19T22:11:21.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.248+0000 7f639ff88700 1 -- 172.21.15.184:0/2596532816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63980f5410 msgr2=0x7f6398108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:21.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.248+0000 7f639ff88700 1 --2- 172.21.15.184:0/2596532816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63980f5410 0x7f6398108a00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f638c005010 tx=0x7f638c00b040).stop 2024-02-19T22:11:21.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.248+0000 7f639ff88700 1 -- 172.21.15.184:0/2596532816 shutdown_connections 2024-02-19T22:11:21.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.248+0000 7f639ff88700 1 --2- 172.21.15.184:0/2596532816 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f638402ff10 0x7f63840323d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:21.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.248+0000 7f639ff88700 1 --2- 172.21.15.184:0/2596532816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63980f5410 0x7f6398108a00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:21.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.248+0000 7f639ff88700 1 -- 172.21.15.184:0/2596532816 >> 172.21.15.184:0/2596532816 conn(0x7f63980f0b70 msgr2=0x7f639805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:21.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.248+0000 7f639ff88700 1 -- 172.21.15.184:0/2596532816 shutdown_connections 2024-02-19T22:11:21.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:21.248+0000 7f639ff88700 1 -- 172.21.15.184:0/2596532816 wait complete. 2024-02-19T22:11:21.250 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:22.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:19.853564+0000 mgr.smithi176.faoogw (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:22.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:21.247925+0000 mon.smithi176 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.184:0/2596532816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:22.859 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:22.860 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:24.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:21.853874+0000 mgr.smithi176.faoogw (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:24.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.621+0000 7f9c27e9f700 1 -- 172.21.15.184:0/2487166999 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c20060280 msgr2=0x7f9c200f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:24.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.621+0000 7f9c27e9f700 1 --2- 172.21.15.184:0/2487166999 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c20060280 0x7f9c200f90f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f9c10009fb0 tx=0x7f9c10009b30).stop 2024-02-19T22:11:24.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.621+0000 7f9c27e9f700 1 -- 172.21.15.184:0/2487166999 shutdown_connections 2024-02-19T22:11:24.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.621+0000 7f9c27e9f700 1 --2- 172.21.15.184:0/2487166999 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c20060280 0x7f9c200f90f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:24.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.621+0000 7f9c27e9f700 1 -- 172.21.15.184:0/2487166999 >> 172.21.15.184:0/2487166999 conn(0x7f9c200f0b70 msgr2=0x7f9c200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:24.625 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.621+0000 7f9c27e9f700 1 -- 172.21.15.184:0/2487166999 shutdown_connections 2024-02-19T22:11:24.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.621+0000 7f9c27e9f700 1 -- 172.21.15.184:0/2487166999 wait complete. 2024-02-19T22:11:24.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.622+0000 7f9c27e9f700 1 Processor -- start 2024-02-19T22:11:24.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.622+0000 7f9c27e9f700 1 -- start start 2024-02-19T22:11:24.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.622+0000 7f9c27e9f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c20060280 0x7f9c200f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:24.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.622+0000 7f9c27e9f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c200f4ff0 con 0x7f9c20060280 2024-02-19T22:11:24.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.622+0000 7f9c25c3b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c20060280 0x7f9c200f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:24.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.622+0000 7f9c25c3b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c20060280 0x7f9c200f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:34414/0 (socket says 172.21.15.184:34414) 2024-02-19T22:11:24.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.622+0000 7f9c25c3b700 1 -- 172.21.15.184:0/1261445451 learned_addr learned my addr 172.21.15.184:0/1261445451 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:24.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.622+0000 7f9c25c3b700 1 -- 172.21.15.184:0/1261445451 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c10005040 con 0x7f9c20060280 2024-02-19T22:11:24.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.622+0000 7f9c25c3b700 1 --2- 172.21.15.184:0/1261445451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c20060280 0x7f9c200f6b70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f9c10009830 tx=0x7f9c1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:24.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.622+0000 7f9c167fc700 1 -- 172.21.15.184:0/1261445451 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c10009320 con 0x7f9c20060280 2024-02-19T22:11:24.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.623+0000 7f9c167fc700 1 -- 172.21.15.184:0/1261445451 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c10013380 con 0x7f9c20060280 2024-02-19T22:11:24.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.623+0000 7f9c27e9f700 1 -- 172.21.15.184:0/1261445451 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c200f5190 con 0x7f9c20060280 2024-02-19T22:11:24.626 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.623+0000 7f9c167fc700 1 -- 172.21.15.184:0/1261445451 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c10014420 con 0x7f9c20060280 2024-02-19T22:11:24.627 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.623+0000 7f9c27e9f700 1 -- 172.21.15.184:0/1261445451 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c200f5620 con 0x7f9c20060280 2024-02-19T22:11:24.627 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.623+0000 7f9c167fc700 1 -- 172.21.15.184:0/1261445451 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c1001e420 con 0x7f9c20060280 2024-02-19T22:11:24.627 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.623+0000 7f9c27e9f700 1 -- 172.21.15.184:0/1261445451 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c20059100 con 0x7f9c20060280 2024-02-19T22:11:24.627 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.624+0000 7f9c167fc700 1 --2- 172.21.15.184:0/1261445451 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9c0c02ff60 0x7f9c0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:24.627 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.624+0000 7f9c167fc700 1 -- 172.21.15.184:0/1261445451 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c10040ee0 con 0x7f9c20060280 2024-02-19T22:11:24.627 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.624+0000 7f9c2543a700 1 --2- 172.21.15.184:0/1261445451 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9c0c02ff60 0x7f9c0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:24.627 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.624+0000 7f9c2543a700 1 --2- 172.21.15.184:0/1261445451 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9c0c02ff60 0x7f9c0c032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f9c1c0096a0 tx=0x7f9c1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:24.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.628+0000 7f9c167fc700 1 -- 172.21.15.184:0/1261445451 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c1001c970 con 0x7f9c20060280 2024-02-19T22:11:24.834 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.833+0000 7f9c27e9f700 1 -- 172.21.15.184:0/1261445451 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c20045ad0 con 0x7f9c20060280 2024-02-19T22:11:24.834 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.833+0000 7f9c167fc700 1 -- 172.21.15.184:0/1261445451 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9c1002c2d0 con 0x7f9c20060280 2024-02-19T22:11:24.835 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:24.835 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:24.836 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.835+0000 7f9c27e9f700 1 -- 172.21.15.184:0/1261445451 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9c0c02ff60 msgr2=0x7f9c0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:24.836 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.835+0000 7f9c27e9f700 1 --2- 172.21.15.184:0/1261445451 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9c0c02ff60 0x7f9c0c032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f9c1c0096a0 tx=0x7f9c1c006b40).stop 2024-02-19T22:11:24.836 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.835+0000 7f9c27e9f700 1 -- 172.21.15.184:0/1261445451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c20060280 msgr2=0x7f9c200f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:24.836 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.835+0000 7f9c27e9f700 1 --2- 172.21.15.184:0/1261445451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c20060280 0x7f9c200f6b70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f9c10009830 tx=0x7f9c1000b040).stop 2024-02-19T22:11:24.836 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.835+0000 7f9c27e9f700 1 -- 172.21.15.184:0/1261445451 shutdown_connections 2024-02-19T22:11:24.836 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.835+0000 7f9c27e9f700 1 --2- 172.21.15.184:0/1261445451 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9c0c02ff60 0x7f9c0c032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:24.836 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.835+0000 7f9c27e9f700 1 --2- 172.21.15.184:0/1261445451 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9c20060280 0x7f9c200f6b70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:24.837 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.835+0000 7f9c27e9f700 1 -- 172.21.15.184:0/1261445451 >> 172.21.15.184:0/1261445451 conn(0x7f9c200f0b70 msgr2=0x7f9c200f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:24.837 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.835+0000 7f9c27e9f700 1 -- 172.21.15.184:0/1261445451 shutdown_connections 2024-02-19T22:11:24.837 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:24.835+0000 7f9c27e9f700 1 -- 172.21.15.184:0/1261445451 wait complete. 2024-02-19T22:11:24.837 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:26.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:23.854288+0000 mgr.smithi176.faoogw (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:26.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:24.834737+0000 mon.smithi176 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.184:0/1261445451' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:26.523 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:26.524 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:28.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:25.854691+0000 mgr.smithi176.faoogw (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:28.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.317+0000 7f638bb13700 1 -- 172.21.15.184:0/3394130910 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63840f6510 msgr2=0x7f63840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:28.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.317+0000 7f638bb13700 1 --2- 172.21.15.184:0/3394130910 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63840f6510 0x7f63840f6930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f6374009fb0 tx=0x7f6374009b30).stop 2024-02-19T22:11:28.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.317+0000 7f638bb13700 1 -- 172.21.15.184:0/3394130910 shutdown_connections 2024-02-19T22:11:28.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.317+0000 7f638bb13700 1 --2- 172.21.15.184:0/3394130910 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63840f6510 0x7f63840f6930 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:28.319 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.317+0000 7f638bb13700 1 -- 172.21.15.184:0/3394130910 >> 172.21.15.184:0/3394130910 conn(0x7f63840f1c70 msgr2=0x7f63840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:28.320 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.318+0000 7f638bb13700 1 -- 172.21.15.184:0/3394130910 shutdown_connections 2024-02-19T22:11:28.320 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.318+0000 7f638bb13700 1 -- 172.21.15.184:0/3394130910 wait complete. 2024-02-19T22:11:28.320 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.318+0000 7f638bb13700 1 Processor -- start 2024-02-19T22:11:28.320 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.318+0000 7f638bb13700 1 -- start start 2024-02-19T22:11:28.320 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.318+0000 7f638bb13700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63840f6510 0x7f63841067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:28.320 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.318+0000 7f638bb13700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6384106cd0 con 0x7f63840f6510 2024-02-19T22:11:28.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.318+0000 7f63898af700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63840f6510 0x7f63841067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:28.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.318+0000 7f63898af700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63840f6510 0x7f63841067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:52072/0 (socket says 172.21.15.184:52072) 2024-02-19T22:11:28.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.318+0000 7f63898af700 1 -- 172.21.15.184:0/1097391596 learned_addr learned my addr 172.21.15.184:0/1097391596 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:28.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.319+0000 7f63898af700 1 -- 172.21.15.184:0/1097391596 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6374005040 con 0x7f63840f6510 2024-02-19T22:11:28.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.319+0000 7f63898af700 1 --2- 172.21.15.184:0/1097391596 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63840f6510 0x7f63841067e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f6374005010 tx=0x7f637400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:28.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.319+0000 7f637a7fc700 1 -- 172.21.15.184:0/1097391596 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6374009320 con 0x7f63840f6510 2024-02-19T22:11:28.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.319+0000 7f638bb13700 1 -- 172.21.15.184:0/1097391596 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6384106ed0 con 0x7f63840f6510 2024-02-19T22:11:28.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.319+0000 7f637a7fc700 1 -- 172.21.15.184:0/1097391596 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6374013380 con 0x7f63840f6510 2024-02-19T22:11:28.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.319+0000 7f637a7fc700 1 -- 172.21.15.184:0/1097391596 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6374014e50 con 0x7f63840f6510 2024-02-19T22:11:28.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.319+0000 7f638bb13700 1 -- 172.21.15.184:0/1097391596 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6384107360 con 0x7f63840f6510 2024-02-19T22:11:28.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.320+0000 7f637a7fc700 1 -- 172.21.15.184:0/1097391596 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63740263a0 con 0x7f63840f6510 2024-02-19T22:11:28.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.320+0000 7f638bb13700 1 -- 172.21.15.184:0/1097391596 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6384045ad0 con 0x7f63840f6510 2024-02-19T22:11:28.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.320+0000 7f637a7fc700 1 --2- 172.21.15.184:0/1097391596 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f637002ff10 0x7f63700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:28.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.320+0000 7f637a7fc700 1 -- 172.21.15.184:0/1097391596 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6374040ea0 con 0x7f63840f6510 2024-02-19T22:11:28.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.320+0000 7f63890ae700 1 --2- 172.21.15.184:0/1097391596 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f637002ff10 0x7f63700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:28.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.321+0000 7f63890ae700 1 --2- 172.21.15.184:0/1097391596 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f637002ff10 0x7f63700323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f63800096a0 tx=0x7f6380006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:28.325 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.324+0000 7f637a7fc700 1 -- 172.21.15.184:0/1097391596 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63740126f0 con 0x7f63840f6510 2024-02-19T22:11:28.529 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.528+0000 7f638bb13700 1 -- 172.21.15.184:0/1097391596 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6384108210 con 0x7f63840f6510 2024-02-19T22:11:28.530 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.529+0000 7f637a7fc700 1 -- 172.21.15.184:0/1097391596 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6374012330 con 0x7f63840f6510 2024-02-19T22:11:28.530 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:28.530 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:28.532 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.530+0000 7f638bb13700 1 -- 172.21.15.184:0/1097391596 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f637002ff10 msgr2=0x7f63700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:28.532 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.530+0000 7f638bb13700 1 --2- 172.21.15.184:0/1097391596 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f637002ff10 0x7f63700323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f63800096a0 tx=0x7f6380006b40).stop 2024-02-19T22:11:28.532 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.530+0000 7f638bb13700 1 -- 172.21.15.184:0/1097391596 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63840f6510 msgr2=0x7f63841067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:28.532 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.530+0000 7f638bb13700 1 --2- 172.21.15.184:0/1097391596 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63840f6510 0x7f63841067e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f6374005010 tx=0x7f637400b040).stop 2024-02-19T22:11:28.532 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.530+0000 7f638bb13700 1 -- 172.21.15.184:0/1097391596 shutdown_connections 2024-02-19T22:11:28.532 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.530+0000 7f638bb13700 1 --2- 172.21.15.184:0/1097391596 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f637002ff10 0x7f63700323d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:28.532 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.530+0000 7f638bb13700 1 --2- 172.21.15.184:0/1097391596 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f63840f6510 0x7f63841067e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:28.532 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.530+0000 7f638bb13700 1 -- 172.21.15.184:0/1097391596 >> 172.21.15.184:0/1097391596 conn(0x7f63840f1c70 msgr2=0x7f63840f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:28.532 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.530+0000 7f638bb13700 1 -- 172.21.15.184:0/1097391596 shutdown_connections 2024-02-19T22:11:28.532 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:28.530+0000 7f638bb13700 1 -- 172.21.15.184:0/1097391596 wait complete. 2024-02-19T22:11:28.533 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:29.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:28 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:28.529936+0000 mon.smithi176 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.184:0/1097391596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:30.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:29 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:27.855084+0000 mgr.smithi176.faoogw (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:30.178 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:30.178 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:31.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.870+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1209789229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f000f51a0 msgr2=0x7f5f000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:31.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.870+0000 7f5f06c9b700 1 --2- 172.21.15.184:0/1209789229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f000f51a0 0x7f5f000f55c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f5efc009fb0 tx=0x7f5efc009b30).stop 2024-02-19T22:11:31.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.870+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1209789229 shutdown_connections 2024-02-19T22:11:31.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.870+0000 7f5f06c9b700 1 --2- 172.21.15.184:0/1209789229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f000f51a0 0x7f5f000f55c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:31.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.870+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1209789229 >> 172.21.15.184:0/1209789229 conn(0x7f5f000f09a0 msgr2=0x7f5f000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:31.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.870+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1209789229 shutdown_connections 2024-02-19T22:11:31.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.870+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1209789229 wait complete. 2024-02-19T22:11:31.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.870+0000 7f5f06c9b700 1 Processor -- start 2024-02-19T22:11:31.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.870+0000 7f5f06c9b700 1 -- start start 2024-02-19T22:11:31.873 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.870+0000 7f5f06c9b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f000f51a0 0x7f5f000f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:31.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.870+0000 7f5f06c9b700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f000f13a0 con 0x7f5f000f51a0 2024-02-19T22:11:31.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.871+0000 7f5f05c99700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f000f51a0 0x7f5f000f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:31.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.871+0000 7f5f05c99700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f000f51a0 0x7f5f000f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:52092/0 (socket says 172.21.15.184:52092) 2024-02-19T22:11:31.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.871+0000 7f5f05c99700 1 -- 172.21.15.184:0/1359344060 learned_addr learned my addr 172.21.15.184:0/1359344060 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:31.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.871+0000 7f5f05c99700 1 -- 172.21.15.184:0/1359344060 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc005040 con 0x7f5f000f51a0 2024-02-19T22:11:31.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.871+0000 7f5f05c99700 1 --2- 172.21.15.184:0/1359344060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f000f51a0 0x7f5f000f2f20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f5efc0089f0 tx=0x7f5efc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:31.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.871+0000 7f5ef67fc700 1 -- 172.21.15.184:0/1359344060 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5efc009320 con 0x7f5f000f51a0 2024-02-19T22:11:31.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.871+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1359344060 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f000f1540 con 0x7f5f000f51a0 2024-02-19T22:11:31.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.871+0000 7f5ef67fc700 1 -- 172.21.15.184:0/1359344060 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5efc013380 con 0x7f5f000f51a0 2024-02-19T22:11:31.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.871+0000 7f5ef67fc700 1 -- 172.21.15.184:0/1359344060 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5efc014e20 con 0x7f5f000f51a0 2024-02-19T22:11:31.874 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.871+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1359344060 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f000f19d0 con 0x7f5f000f51a0 2024-02-19T22:11:31.875 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.872+0000 7f5ef67fc700 1 -- 172.21.15.184:0/1359344060 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5efc0095f0 con 0x7f5f000f51a0 2024-02-19T22:11:31.875 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.872+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1359344060 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f00059100 con 0x7f5f000f51a0 2024-02-19T22:11:31.875 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.872+0000 7f5ef67fc700 1 --2- 172.21.15.184:0/1359344060 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f5eec02ff10 0x7f5eec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:31.875 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.872+0000 7f5ef67fc700 1 -- 172.21.15.184:0/1359344060 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5efc040f50 con 0x7f5f000f51a0 2024-02-19T22:11:31.875 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.873+0000 7f5f05498700 1 --2- 172.21.15.184:0/1359344060 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f5eec02ff10 0x7f5eec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:31.875 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.873+0000 7f5f05498700 1 --2- 172.21.15.184:0/1359344060 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f5eec02ff10 0x7f5eec0323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f5ef00096a0 tx=0x7f5ef0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:31.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:31.877+0000 7f5ef67fc700 1 -- 172.21.15.184:0/1359344060 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5efc014420 con 0x7f5f000f51a0 2024-02-19T22:11:32.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:32.082+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1359344060 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f00045ad0 con 0x7f5f000f51a0 2024-02-19T22:11:32.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:32.082+0000 7f5ef67fc700 1 -- 172.21.15.184:0/1359344060 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5efc01d9d0 con 0x7f5f000f51a0 2024-02-19T22:11:32.084 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:32.084 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:32.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:32.084+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1359344060 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f5eec02ff10 msgr2=0x7f5eec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:32.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:32.084+0000 7f5f06c9b700 1 --2- 172.21.15.184:0/1359344060 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f5eec02ff10 0x7f5eec0323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f5ef00096a0 tx=0x7f5ef0006b40).stop 2024-02-19T22:11:32.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:32.084+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1359344060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f000f51a0 msgr2=0x7f5f000f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:32.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:32.084+0000 7f5f06c9b700 1 --2- 172.21.15.184:0/1359344060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f000f51a0 0x7f5f000f2f20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f5efc0089f0 tx=0x7f5efc00b040).stop 2024-02-19T22:11:32.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:32.084+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1359344060 shutdown_connections 2024-02-19T22:11:32.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:32.084+0000 7f5f06c9b700 1 --2- 172.21.15.184:0/1359344060 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f5eec02ff10 0x7f5eec0323d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:32.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:32.084+0000 7f5f06c9b700 1 --2- 172.21.15.184:0/1359344060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5f000f51a0 0x7f5f000f2f20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:32.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:32.084+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1359344060 >> 172.21.15.184:0/1359344060 conn(0x7f5f000f09a0 msgr2=0x7f5f0005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:32.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:32.084+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1359344060 shutdown_connections 2024-02-19T22:11:32.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:32.084+0000 7f5f06c9b700 1 -- 172.21.15.184:0/1359344060 wait complete. 2024-02-19T22:11:32.086 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:32.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:29.855408+0000 mgr.smithi176.faoogw (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:33.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:32 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:32.083724+0000 mon.smithi176 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.184:0/1359344060' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:33.609 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:33.609 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:34.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:31.855633+0000 mgr.smithi176.faoogw (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:35.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.489+0000 7fefc1509700 1 -- 172.21.15.184:0/906058358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefbc060280 msgr2=0x7fefbc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:35.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.489+0000 7fefc1509700 1 --2- 172.21.15.184:0/906058358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefbc060280 0x7fefbc0f90f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fefac009fb0 tx=0x7fefac009b30).stop 2024-02-19T22:11:35.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.489+0000 7fefc1509700 1 -- 172.21.15.184:0/906058358 shutdown_connections 2024-02-19T22:11:35.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.489+0000 7fefc1509700 1 --2- 172.21.15.184:0/906058358 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefbc060280 0x7fefbc0f90f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:35.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.489+0000 7fefc1509700 1 -- 172.21.15.184:0/906058358 >> 172.21.15.184:0/906058358 conn(0x7fefbc0f0b70 msgr2=0x7fefbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:35.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.489+0000 7fefc1509700 1 -- 172.21.15.184:0/906058358 shutdown_connections 2024-02-19T22:11:35.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.489+0000 7fefc1509700 1 -- 172.21.15.184:0/906058358 wait complete. 2024-02-19T22:11:35.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.489+0000 7fefc1509700 1 Processor -- start 2024-02-19T22:11:35.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.489+0000 7fefc1509700 1 -- start start 2024-02-19T22:11:35.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.490+0000 7fefc1509700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefbc060280 0x7fefbc1067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:35.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.490+0000 7fefc1509700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc106c90 con 0x7fefbc060280 2024-02-19T22:11:35.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.490+0000 7fefbaffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefbc060280 0x7fefbc1067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:35.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.490+0000 7fefbaffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefbc060280 0x7fefbc1067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:52100/0 (socket says 172.21.15.184:52100) 2024-02-19T22:11:35.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.490+0000 7fefbaffd700 1 -- 172.21.15.184:0/3709916530 learned_addr learned my addr 172.21.15.184:0/3709916530 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:35.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.490+0000 7fefbaffd700 1 -- 172.21.15.184:0/3709916530 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac005040 con 0x7fefbc060280 2024-02-19T22:11:35.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.490+0000 7fefbaffd700 1 --2- 172.21.15.184:0/3709916530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefbc060280 0x7fefbc1067a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fefac0089f0 tx=0x7fefac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:35.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.490+0000 7fefa37fe700 1 -- 172.21.15.184:0/3709916530 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefac009320 con 0x7fefbc060280 2024-02-19T22:11:35.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.490+0000 7fefa37fe700 1 -- 172.21.15.184:0/3709916530 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefac013380 con 0x7fefbc060280 2024-02-19T22:11:35.494 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.490+0000 7fefa37fe700 1 -- 172.21.15.184:0/3709916530 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefac014e20 con 0x7fefbc060280 2024-02-19T22:11:35.494 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.490+0000 7fefc1509700 1 -- 172.21.15.184:0/3709916530 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefbc106e90 con 0x7fefbc060280 2024-02-19T22:11:35.494 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.491+0000 7fefc1509700 1 -- 172.21.15.184:0/3709916530 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefbc107320 con 0x7fefbc060280 2024-02-19T22:11:35.494 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.491+0000 7fefc1509700 1 -- 172.21.15.184:0/3709916530 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefbc045ad0 con 0x7fefbc060280 2024-02-19T22:11:35.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.491+0000 7fefa37fe700 1 -- 172.21.15.184:0/3709916530 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefac0095f0 con 0x7fefbc060280 2024-02-19T22:11:35.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.492+0000 7fefa37fe700 1 --2- 172.21.15.184:0/3709916530 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fefa402ff10 0x7fefa40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:35.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.492+0000 7fefa37fe700 1 -- 172.21.15.184:0/3709916530 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefac040e90 con 0x7fefbc060280 2024-02-19T22:11:35.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.492+0000 7fefba7fc700 1 --2- 172.21.15.184:0/3709916530 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fefa402ff10 0x7fefa40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:35.495 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.492+0000 7fefba7fc700 1 --2- 172.21.15.184:0/3709916530 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fefa402ff10 0x7fefa40323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fefb00096a0 tx=0x7fefb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:35.497 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.496+0000 7fefa37fe700 1 -- 172.21.15.184:0/3709916530 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefac014420 con 0x7fefbc060280 2024-02-19T22:11:35.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.704+0000 7fefc1509700 1 -- 172.21.15.184:0/3709916530 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefbc1081d0 con 0x7fefbc060280 2024-02-19T22:11:35.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.705+0000 7fefa37fe700 1 -- 172.21.15.184:0/3709916530 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fefac01d980 con 0x7fefbc060280 2024-02-19T22:11:35.706 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:35.706 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:35.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.706+0000 7fefc1509700 1 -- 172.21.15.184:0/3709916530 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fefa402ff10 msgr2=0x7fefa40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:35.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.706+0000 7fefc1509700 1 --2- 172.21.15.184:0/3709916530 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fefa402ff10 0x7fefa40323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fefb00096a0 tx=0x7fefb0006b40).stop 2024-02-19T22:11:35.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.706+0000 7fefc1509700 1 -- 172.21.15.184:0/3709916530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefbc060280 msgr2=0x7fefbc1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:35.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.706+0000 7fefc1509700 1 --2- 172.21.15.184:0/3709916530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefbc060280 0x7fefbc1067a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fefac0089f0 tx=0x7fefac00b040).stop 2024-02-19T22:11:35.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.706+0000 7fefc1509700 1 -- 172.21.15.184:0/3709916530 shutdown_connections 2024-02-19T22:11:35.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.706+0000 7fefc1509700 1 --2- 172.21.15.184:0/3709916530 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fefa402ff10 0x7fefa40323d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:35.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.706+0000 7fefc1509700 1 --2- 172.21.15.184:0/3709916530 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefbc060280 0x7fefbc1067a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:35.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.706+0000 7fefc1509700 1 -- 172.21.15.184:0/3709916530 >> 172.21.15.184:0/3709916530 conn(0x7fefbc0f0b70 msgr2=0x7fefbc0f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:35.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.706+0000 7fefc1509700 1 -- 172.21.15.184:0/3709916530 shutdown_connections 2024-02-19T22:11:35.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:35.706+0000 7fefc1509700 1 -- 172.21.15.184:0/3709916530 wait complete. 2024-02-19T22:11:35.708 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:36.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:33.855962+0000 mgr.smithi176.faoogw (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:36.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:35.705945+0000 mon.smithi176 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.184:0/3709916530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:35.968672+0000 mon.smithi176 (mon.0) 160 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:11:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:35.971497+0000 mon.smithi176 (mon.0) 161 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:11:37.376 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:37.376 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:38.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:35.856309+0000 mgr.smithi176.faoogw (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:39.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.095+0000 7fa8485fb700 1 -- 172.21.15.184:0/1202291804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8400f5410 msgr2=0x7fa8400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:39.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.095+0000 7fa8485fb700 1 --2- 172.21.15.184:0/1202291804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8400f5410 0x7fa8400f5830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fa830009fb0 tx=0x7fa830009b30).stop 2024-02-19T22:11:39.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.095+0000 7fa8485fb700 1 -- 172.21.15.184:0/1202291804 shutdown_connections 2024-02-19T22:11:39.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.095+0000 7fa8485fb700 1 --2- 172.21.15.184:0/1202291804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8400f5410 0x7fa8400f5830 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:39.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.095+0000 7fa8485fb700 1 -- 172.21.15.184:0/1202291804 >> 172.21.15.184:0/1202291804 conn(0x7fa8400f0b70 msgr2=0x7fa8400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:39.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.095+0000 7fa8485fb700 1 -- 172.21.15.184:0/1202291804 shutdown_connections 2024-02-19T22:11:39.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.095+0000 7fa8485fb700 1 -- 172.21.15.184:0/1202291804 wait complete. 2024-02-19T22:11:39.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.095+0000 7fa8485fb700 1 Processor -- start 2024-02-19T22:11:39.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.095+0000 7fa8485fb700 1 -- start start 2024-02-19T22:11:39.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.095+0000 7fa8485fb700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8400f5410 0x7fa840109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:39.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.095+0000 7fa8485fb700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa840060220 con 0x7fa8400f5410 2024-02-19T22:11:39.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.096+0000 7fa846397700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8400f5410 0x7fa840109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:39.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.096+0000 7fa846397700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8400f5410 0x7fa840109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:49422/0 (socket says 172.21.15.184:49422) 2024-02-19T22:11:39.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.096+0000 7fa846397700 1 -- 172.21.15.184:0/811350510 learned_addr learned my addr 172.21.15.184:0/811350510 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:39.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.096+0000 7fa846397700 1 -- 172.21.15.184:0/811350510 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa830005040 con 0x7fa8400f5410 2024-02-19T22:11:39.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.096+0000 7fa846397700 1 --2- 172.21.15.184:0/811350510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8400f5410 0x7fa840109540 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa830005010 tx=0x7fa83000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:39.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.096+0000 7fa836ffd700 1 -- 172.21.15.184:0/811350510 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa830009320 con 0x7fa8400f5410 2024-02-19T22:11:39.099 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.096+0000 7fa836ffd700 1 -- 172.21.15.184:0/811350510 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa830013380 con 0x7fa8400f5410 2024-02-19T22:11:39.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.096+0000 7fa8485fb700 1 -- 172.21.15.184:0/811350510 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa840060420 con 0x7fa8400f5410 2024-02-19T22:11:39.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.096+0000 7fa836ffd700 1 -- 172.21.15.184:0/811350510 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa830014e50 con 0x7fa8400f5410 2024-02-19T22:11:39.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.096+0000 7fa8485fb700 1 -- 172.21.15.184:0/811350510 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa840109bf0 con 0x7fa8400f5410 2024-02-19T22:11:39.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.097+0000 7fa836ffd700 1 -- 172.21.15.184:0/811350510 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8300263a0 con 0x7fa8400f5410 2024-02-19T22:11:39.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.097+0000 7fa8485fb700 1 -- 172.21.15.184:0/811350510 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa840059100 con 0x7fa8400f5410 2024-02-19T22:11:39.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.097+0000 7fa836ffd700 1 --2- 172.21.15.184:0/811350510 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa82c02ff10 0x7fa82c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:39.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.097+0000 7fa836ffd700 1 -- 172.21.15.184:0/811350510 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8300411b0 con 0x7fa8400f5410 2024-02-19T22:11:39.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.098+0000 7fa845b96700 1 --2- 172.21.15.184:0/811350510 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa82c02ff10 0x7fa82c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:39.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.098+0000 7fa845b96700 1 --2- 172.21.15.184:0/811350510 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa82c02ff10 0x7fa82c0323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fa83c0096a0 tx=0x7fa83c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:39.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.102+0000 7fa836ffd700 1 -- 172.21.15.184:0/811350510 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8300126f0 con 0x7fa8400f5410 2024-02-19T22:11:39.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:38 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:37.856681+0000 mgr.smithi176.faoogw (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:39.305 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.304+0000 7fa8485fb700 1 -- 172.21.15.184:0/811350510 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa840045ad0 con 0x7fa8400f5410 2024-02-19T22:11:39.306 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.305+0000 7fa836ffd700 1 -- 172.21.15.184:0/811350510 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa83000f340 con 0x7fa8400f5410 2024-02-19T22:11:39.306 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:39.306 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:39.308 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.306+0000 7fa8485fb700 1 -- 172.21.15.184:0/811350510 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa82c02ff10 msgr2=0x7fa82c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:39.308 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.306+0000 7fa8485fb700 1 --2- 172.21.15.184:0/811350510 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa82c02ff10 0x7fa82c0323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fa83c0096a0 tx=0x7fa83c006b40).stop 2024-02-19T22:11:39.308 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.306+0000 7fa8485fb700 1 -- 172.21.15.184:0/811350510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8400f5410 msgr2=0x7fa840109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:39.308 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.306+0000 7fa8485fb700 1 --2- 172.21.15.184:0/811350510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8400f5410 0x7fa840109540 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa830005010 tx=0x7fa83000b040).stop 2024-02-19T22:11:39.308 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.306+0000 7fa8485fb700 1 -- 172.21.15.184:0/811350510 shutdown_connections 2024-02-19T22:11:39.308 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.307+0000 7fa8485fb700 1 --2- 172.21.15.184:0/811350510 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa82c02ff10 0x7fa82c0323d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:39.308 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.307+0000 7fa8485fb700 1 --2- 172.21.15.184:0/811350510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8400f5410 0x7fa840109540 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:39.308 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.307+0000 7fa8485fb700 1 -- 172.21.15.184:0/811350510 >> 172.21.15.184:0/811350510 conn(0x7fa8400f0b70 msgr2=0x7fa8400f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:39.308 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.307+0000 7fa8485fb700 1 -- 172.21.15.184:0/811350510 shutdown_connections 2024-02-19T22:11:39.309 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:39.307+0000 7fa8485fb700 1 -- 172.21.15.184:0/811350510 wait complete. 2024-02-19T22:11:39.309 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:40.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:39.306365+0000 mon.smithi176 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.184:0/811350510' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:40.929 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:40.929 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:41.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:40 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:39.856986+0000 mgr.smithi176.faoogw (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:42.715 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.711+0000 7f058a6b2700 1 -- 172.21.15.184:0/2763287513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0584060280 msgr2=0x7f05840f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:42.715 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.711+0000 7f058a6b2700 1 --2- 172.21.15.184:0/2763287513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0584060280 0x7f05840f90f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f0578009fb0 tx=0x7f0578009b30).stop 2024-02-19T22:11:42.715 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.711+0000 7f058a6b2700 1 -- 172.21.15.184:0/2763287513 shutdown_connections 2024-02-19T22:11:42.715 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.711+0000 7f058a6b2700 1 --2- 172.21.15.184:0/2763287513 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0584060280 0x7f05840f90f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:42.715 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.711+0000 7f058a6b2700 1 -- 172.21.15.184:0/2763287513 >> 172.21.15.184:0/2763287513 conn(0x7f05840f0b70 msgr2=0x7f05840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:42.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.711+0000 7f058a6b2700 1 -- 172.21.15.184:0/2763287513 shutdown_connections 2024-02-19T22:11:42.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.711+0000 7f058a6b2700 1 -- 172.21.15.184:0/2763287513 wait complete. 2024-02-19T22:11:42.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.711+0000 7f058a6b2700 1 Processor -- start 2024-02-19T22:11:42.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.712+0000 7f058a6b2700 1 -- start start 2024-02-19T22:11:42.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.712+0000 7f058a6b2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0584060280 0x7f0584106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:42.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.712+0000 7f058a6b2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0584106c70 con 0x7f0584060280 2024-02-19T22:11:42.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.712+0000 7f0583fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0584060280 0x7f0584106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:42.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.712+0000 7f0583fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0584060280 0x7f0584106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:49446/0 (socket says 172.21.15.184:49446) 2024-02-19T22:11:42.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.712+0000 7f0583fff700 1 -- 172.21.15.184:0/3340315863 learned_addr learned my addr 172.21.15.184:0/3340315863 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:42.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.712+0000 7f0583fff700 1 -- 172.21.15.184:0/3340315863 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0578005040 con 0x7f0584060280 2024-02-19T22:11:42.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.712+0000 7f0583fff700 1 --2- 172.21.15.184:0/3340315863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0584060280 0x7f0584106780 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f05780089f0 tx=0x7f057800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:42.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.713+0000 7f0580ff9700 1 -- 172.21.15.184:0/3340315863 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0578009320 con 0x7f0584060280 2024-02-19T22:11:42.717 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.713+0000 7f0580ff9700 1 -- 172.21.15.184:0/3340315863 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0578013380 con 0x7f0584060280 2024-02-19T22:11:42.717 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.713+0000 7f058a6b2700 1 -- 172.21.15.184:0/3340315863 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0584106e70 con 0x7f0584060280 2024-02-19T22:11:42.717 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.713+0000 7f0580ff9700 1 -- 172.21.15.184:0/3340315863 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0578014e20 con 0x7f0584060280 2024-02-19T22:11:42.717 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.713+0000 7f058a6b2700 1 -- 172.21.15.184:0/3340315863 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0584107300 con 0x7f0584060280 2024-02-19T22:11:42.717 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.713+0000 7f0580ff9700 1 -- 172.21.15.184:0/3340315863 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f05780095f0 con 0x7f0584060280 2024-02-19T22:11:42.717 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.714+0000 7f058a6b2700 1 -- 172.21.15.184:0/3340315863 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0584045ad0 con 0x7f0584060280 2024-02-19T22:11:42.717 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.714+0000 7f0580ff9700 1 --2- 172.21.15.184:0/3340315863 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f056c02ff10 0x7f056c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:42.717 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.714+0000 7f0580ff9700 1 -- 172.21.15.184:0/3340315863 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0578040f50 con 0x7f0584060280 2024-02-19T22:11:42.717 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.714+0000 7f05837fe700 1 --2- 172.21.15.184:0/3340315863 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f056c02ff10 0x7f056c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:42.717 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.714+0000 7f05837fe700 1 --2- 172.21.15.184:0/3340315863 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f056c02ff10 0x7f056c0323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f05740096a0 tx=0x7f0574006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:42.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.719+0000 7f0580ff9700 1 -- 172.21.15.184:0/3340315863 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0578014420 con 0x7f0584060280 2024-02-19T22:11:42.922 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.921+0000 7f058a6b2700 1 -- 172.21.15.184:0/3340315863 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05841081b0 con 0x7f0584060280 2024-02-19T22:11:42.923 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.921+0000 7f0580ff9700 1 -- 172.21.15.184:0/3340315863 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f057801d980 con 0x7f0584060280 2024-02-19T22:11:42.923 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:42.923 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:42.924 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.923+0000 7f058a6b2700 1 -- 172.21.15.184:0/3340315863 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f056c02ff10 msgr2=0x7f056c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:42.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.923+0000 7f058a6b2700 1 --2- 172.21.15.184:0/3340315863 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f056c02ff10 0x7f056c0323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f05740096a0 tx=0x7f0574006b40).stop 2024-02-19T22:11:42.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.923+0000 7f058a6b2700 1 -- 172.21.15.184:0/3340315863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0584060280 msgr2=0x7f0584106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:42.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.923+0000 7f058a6b2700 1 --2- 172.21.15.184:0/3340315863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0584060280 0x7f0584106780 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f05780089f0 tx=0x7f057800b040).stop 2024-02-19T22:11:42.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.923+0000 7f058a6b2700 1 -- 172.21.15.184:0/3340315863 shutdown_connections 2024-02-19T22:11:42.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.923+0000 7f058a6b2700 1 --2- 172.21.15.184:0/3340315863 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f056c02ff10 0x7f056c0323d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:42.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.923+0000 7f058a6b2700 1 --2- 172.21.15.184:0/3340315863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0584060280 0x7f0584106780 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:42.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.923+0000 7f058a6b2700 1 -- 172.21.15.184:0/3340315863 >> 172.21.15.184:0/3340315863 conn(0x7f05840f0b70 msgr2=0x7f05840f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:42.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.923+0000 7f058a6b2700 1 -- 172.21.15.184:0/3340315863 shutdown_connections 2024-02-19T22:11:42.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:42.923+0000 7f058a6b2700 1 -- 172.21.15.184:0/3340315863 wait complete. 2024-02-19T22:11:42.926 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:43.385 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:42 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:41.857384+0000 mgr.smithi176.faoogw (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:44.385 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:42.922831+0000 mon.smithi176 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.184:0/3340315863' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:44.589 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:44.589 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:46.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:43.857665+0000 mgr.smithi176.faoogw (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:46.482 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.477+0000 7f43c1629700 1 -- 172.21.15.184:0/1176168715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43bc0f5410 msgr2=0x7f43bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:46.482 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.477+0000 7f43c1629700 1 --2- 172.21.15.184:0/1176168715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43bc0f5410 0x7f43bc0f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f43b0009fb0 tx=0x7f43b0009b30).stop 2024-02-19T22:11:46.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.477+0000 7f43c1629700 1 -- 172.21.15.184:0/1176168715 shutdown_connections 2024-02-19T22:11:46.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.477+0000 7f43c1629700 1 --2- 172.21.15.184:0/1176168715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43bc0f5410 0x7f43bc0f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:46.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.477+0000 7f43c1629700 1 -- 172.21.15.184:0/1176168715 >> 172.21.15.184:0/1176168715 conn(0x7f43bc0f0b70 msgr2=0x7f43bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:46.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.477+0000 7f43c1629700 1 -- 172.21.15.184:0/1176168715 shutdown_connections 2024-02-19T22:11:46.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.477+0000 7f43c1629700 1 -- 172.21.15.184:0/1176168715 wait complete. 2024-02-19T22:11:46.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.478+0000 7f43c1629700 1 Processor -- start 2024-02-19T22:11:46.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.478+0000 7f43c1629700 1 -- start start 2024-02-19T22:11:46.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.478+0000 7f43c1629700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43bc0f5410 0x7f43bc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:46.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.478+0000 7f43c1629700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43bc105bd0 con 0x7f43bc0f5410 2024-02-19T22:11:46.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.479+0000 7f43baffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43bc0f5410 0x7f43bc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:46.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.479+0000 7f43baffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43bc0f5410 0x7f43bc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:42896/0 (socket says 172.21.15.184:42896) 2024-02-19T22:11:46.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.479+0000 7f43baffd700 1 -- 172.21.15.184:0/2357266799 learned_addr learned my addr 172.21.15.184:0/2357266799 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:46.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.479+0000 7f43baffd700 1 -- 172.21.15.184:0/2357266799 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b0005040 con 0x7f43bc0f5410 2024-02-19T22:11:46.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.479+0000 7f43baffd700 1 --2- 172.21.15.184:0/2357266799 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43bc0f5410 0x7f43bc1056e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f43b0005010 tx=0x7f43b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:46.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.479+0000 7f43a37fe700 1 -- 172.21.15.184:0/2357266799 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43b0009320 con 0x7f43bc0f5410 2024-02-19T22:11:46.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.479+0000 7f43a37fe700 1 -- 172.21.15.184:0/2357266799 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f43b0013380 con 0x7f43bc0f5410 2024-02-19T22:11:46.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.479+0000 7f43c1629700 1 -- 172.21.15.184:0/2357266799 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43bc105dd0 con 0x7f43bc0f5410 2024-02-19T22:11:46.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.479+0000 7f43a37fe700 1 -- 172.21.15.184:0/2357266799 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43b0014e50 con 0x7f43bc0f5410 2024-02-19T22:11:46.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.479+0000 7f43c1629700 1 -- 172.21.15.184:0/2357266799 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43bc106260 con 0x7f43bc0f5410 2024-02-19T22:11:46.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.480+0000 7f43a37fe700 1 -- 172.21.15.184:0/2357266799 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f43b0014420 con 0x7f43bc0f5410 2024-02-19T22:11:46.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.480+0000 7f43c1629700 1 -- 172.21.15.184:0/2357266799 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43bc045ad0 con 0x7f43bc0f5410 2024-02-19T22:11:46.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.480+0000 7f43a37fe700 1 --2- 172.21.15.184:0/2357266799 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f43a402ff10 0x7f43a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:46.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.480+0000 7f43a37fe700 1 -- 172.21.15.184:0/2357266799 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f43b0040e30 con 0x7f43bc0f5410 2024-02-19T22:11:46.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.481+0000 7f43ba7fc700 1 --2- 172.21.15.184:0/2357266799 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f43a402ff10 0x7f43a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:46.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.481+0000 7f43ba7fc700 1 --2- 172.21.15.184:0/2357266799 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f43a402ff10 0x7f43a40323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f43ac0096a0 tx=0x7f43ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:46.486 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.485+0000 7f43a37fe700 1 -- 172.21.15.184:0/2357266799 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43b00095f0 con 0x7f43bc0f5410 2024-02-19T22:11:46.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.692+0000 7f43c1629700 1 -- 172.21.15.184:0/2357266799 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43bc107110 con 0x7f43bc0f5410 2024-02-19T22:11:46.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.693+0000 7f43a37fe700 1 -- 172.21.15.184:0/2357266799 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f43b001db70 con 0x7f43bc0f5410 2024-02-19T22:11:46.694 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:46.694 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:46.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.694+0000 7f43c1629700 1 -- 172.21.15.184:0/2357266799 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f43a402ff10 msgr2=0x7f43a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:46.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.694+0000 7f43c1629700 1 --2- 172.21.15.184:0/2357266799 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f43a402ff10 0x7f43a40323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f43ac0096a0 tx=0x7f43ac006b40).stop 2024-02-19T22:11:46.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.694+0000 7f43c1629700 1 -- 172.21.15.184:0/2357266799 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43bc0f5410 msgr2=0x7f43bc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:46.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.694+0000 7f43c1629700 1 --2- 172.21.15.184:0/2357266799 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43bc0f5410 0x7f43bc1056e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f43b0005010 tx=0x7f43b000b040).stop 2024-02-19T22:11:46.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.695+0000 7f43c1629700 1 -- 172.21.15.184:0/2357266799 shutdown_connections 2024-02-19T22:11:46.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.695+0000 7f43c1629700 1 --2- 172.21.15.184:0/2357266799 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f43a402ff10 0x7f43a40323d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:46.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.695+0000 7f43c1629700 1 --2- 172.21.15.184:0/2357266799 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43bc0f5410 0x7f43bc1056e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:46.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.695+0000 7f43c1629700 1 -- 172.21.15.184:0/2357266799 >> 172.21.15.184:0/2357266799 conn(0x7f43bc0f0b70 msgr2=0x7f43bc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:46.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.695+0000 7f43c1629700 1 -- 172.21.15.184:0/2357266799 shutdown_connections 2024-02-19T22:11:46.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:46.695+0000 7f43c1629700 1 -- 172.21.15.184:0/2357266799 wait complete. 2024-02-19T22:11:46.697 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:47.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:46 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:46.694186+0000 mon.smithi176 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.184:0/2357266799' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:48.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:45.857985+0000 mgr.smithi176.faoogw (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:48.357 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:48.357 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:50.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:49 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:47.858263+0000 mgr.smithi176.faoogw (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:50.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.228+0000 7fe23d59f700 1 -- 172.21.15.184:0/53520917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2380f5410 msgr2=0x7fe2380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:50.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.228+0000 7fe23d59f700 1 --2- 172.21.15.184:0/53520917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2380f5410 0x7fe2380f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fe22c009fb0 tx=0x7fe22c009b30).stop 2024-02-19T22:11:50.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.228+0000 7fe23d59f700 1 -- 172.21.15.184:0/53520917 shutdown_connections 2024-02-19T22:11:50.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.228+0000 7fe23d59f700 1 --2- 172.21.15.184:0/53520917 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2380f5410 0x7fe2380f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:50.231 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.228+0000 7fe23d59f700 1 -- 172.21.15.184:0/53520917 >> 172.21.15.184:0/53520917 conn(0x7fe2380f0b70 msgr2=0x7fe2380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:50.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.228+0000 7fe23d59f700 1 -- 172.21.15.184:0/53520917 shutdown_connections 2024-02-19T22:11:50.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.228+0000 7fe23d59f700 1 -- 172.21.15.184:0/53520917 wait complete. 2024-02-19T22:11:50.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.229+0000 7fe23d59f700 1 Processor -- start 2024-02-19T22:11:50.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.229+0000 7fe23d59f700 1 -- start start 2024-02-19T22:11:50.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.229+0000 7fe23d59f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2380f5410 0x7fe2380fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:50.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.229+0000 7fe23d59f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2380fb960 con 0x7fe2380f5410 2024-02-19T22:11:50.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.229+0000 7fe236ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2380f5410 0x7fe2380fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:50.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.229+0000 7fe236ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2380f5410 0x7fe2380fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:42910/0 (socket says 172.21.15.184:42910) 2024-02-19T22:11:50.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.229+0000 7fe236ffd700 1 -- 172.21.15.184:0/4248481382 learned_addr learned my addr 172.21.15.184:0/4248481382 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:50.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.230+0000 7fe236ffd700 1 -- 172.21.15.184:0/4248481382 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe22c005040 con 0x7fe2380f5410 2024-02-19T22:11:50.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.230+0000 7fe236ffd700 1 --2- 172.21.15.184:0/4248481382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2380f5410 0x7fe2380fd4e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fe22c0089f0 tx=0x7fe22c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:50.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.230+0000 7fe21f7fe700 1 -- 172.21.15.184:0/4248481382 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe22c009320 con 0x7fe2380f5410 2024-02-19T22:11:50.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.230+0000 7fe23d59f700 1 -- 172.21.15.184:0/4248481382 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2380fbb00 con 0x7fe2380f5410 2024-02-19T22:11:50.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.230+0000 7fe21f7fe700 1 -- 172.21.15.184:0/4248481382 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe22c013380 con 0x7fe2380f5410 2024-02-19T22:11:50.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.230+0000 7fe21f7fe700 1 -- 172.21.15.184:0/4248481382 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe22c01ed80 con 0x7fe2380f5410 2024-02-19T22:11:50.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.230+0000 7fe23d59f700 1 -- 172.21.15.184:0/4248481382 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2380fbf90 con 0x7fe2380f5410 2024-02-19T22:11:50.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.231+0000 7fe21f7fe700 1 -- 172.21.15.184:0/4248481382 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe22c01e420 con 0x7fe2380f5410 2024-02-19T22:11:50.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.231+0000 7fe23d59f700 1 -- 172.21.15.184:0/4248481382 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe238059100 con 0x7fe2380f5410 2024-02-19T22:11:50.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.231+0000 7fe21f7fe700 1 --2- 172.21.15.184:0/4248481382 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe22002ff60 0x7fe220032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:50.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.231+0000 7fe21f7fe700 1 -- 172.21.15.184:0/4248481382 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe22c041030 con 0x7fe2380f5410 2024-02-19T22:11:50.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.231+0000 7fe2367fc700 1 --2- 172.21.15.184:0/4248481382 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe22002ff60 0x7fe220032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:50.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.231+0000 7fe2367fc700 1 --2- 172.21.15.184:0/4248481382 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe22002ff60 0x7fe220032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fe2280096a0 tx=0x7fe228006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:50.237 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.236+0000 7fe21f7fe700 1 -- 172.21.15.184:0/4248481382 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe22c01ccf0 con 0x7fe2380f5410 2024-02-19T22:11:50.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.442+0000 7fe23d59f700 1 -- 172.21.15.184:0/4248481382 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe238045ad0 con 0x7fe2380f5410 2024-02-19T22:11:50.444 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.443+0000 7fe21f7fe700 1 -- 172.21.15.184:0/4248481382 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe22c01cb10 con 0x7fe2380f5410 2024-02-19T22:11:50.444 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:50.444 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:50.445 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.444+0000 7fe23d59f700 1 -- 172.21.15.184:0/4248481382 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe22002ff60 msgr2=0x7fe220032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:50.445 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.444+0000 7fe23d59f700 1 --2- 172.21.15.184:0/4248481382 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe22002ff60 0x7fe220032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fe2280096a0 tx=0x7fe228006b40).stop 2024-02-19T22:11:50.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.444+0000 7fe23d59f700 1 -- 172.21.15.184:0/4248481382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2380f5410 msgr2=0x7fe2380fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:50.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.444+0000 7fe23d59f700 1 --2- 172.21.15.184:0/4248481382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2380f5410 0x7fe2380fd4e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fe22c0089f0 tx=0x7fe22c00b040).stop 2024-02-19T22:11:50.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.444+0000 7fe23d59f700 1 -- 172.21.15.184:0/4248481382 shutdown_connections 2024-02-19T22:11:50.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.444+0000 7fe23d59f700 1 --2- 172.21.15.184:0/4248481382 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe22002ff60 0x7fe220032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:50.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.444+0000 7fe23d59f700 1 --2- 172.21.15.184:0/4248481382 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe2380f5410 0x7fe2380fd4e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:50.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.444+0000 7fe23d59f700 1 -- 172.21.15.184:0/4248481382 >> 172.21.15.184:0/4248481382 conn(0x7fe2380f0b70 msgr2=0x7fe23805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:50.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.444+0000 7fe23d59f700 1 -- 172.21.15.184:0/4248481382 shutdown_connections 2024-02-19T22:11:50.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:50.444+0000 7fe23d59f700 1 -- 172.21.15.184:0/4248481382 wait complete. 2024-02-19T22:11:50.446 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:51.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:50 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:50.443970+0000 mon.smithi176 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.184:0/4248481382' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:52.051 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:52.052 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:52.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:51 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:49.858541+0000 mgr.smithi176.faoogw (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:53.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.942+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/935060306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e740f5410 msgr2=0x7f2e740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:53.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.942+0000 7f2e7bb4c700 1 --2- 172.21.15.184:0/935060306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e740f5410 0x7f2e740f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f2e64009fb0 tx=0x7f2e64009b30).stop 2024-02-19T22:11:53.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.942+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/935060306 shutdown_connections 2024-02-19T22:11:53.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.942+0000 7f2e7bb4c700 1 --2- 172.21.15.184:0/935060306 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e740f5410 0x7f2e740f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:53.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.942+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/935060306 >> 172.21.15.184:0/935060306 conn(0x7f2e740f0b70 msgr2=0x7f2e740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:53.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.942+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/935060306 shutdown_connections 2024-02-19T22:11:53.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.942+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/935060306 wait complete. 2024-02-19T22:11:53.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.943+0000 7f2e7bb4c700 1 Processor -- start 2024-02-19T22:11:53.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.943+0000 7f2e7bb4c700 1 -- start start 2024-02-19T22:11:53.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.943+0000 7f2e7bb4c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e740f5410 0x7f2e74109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:53.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.943+0000 7f2e7bb4c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e74060220 con 0x7f2e740f5410 2024-02-19T22:11:53.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.943+0000 7f2e798e8700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e740f5410 0x7f2e74109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:53.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.943+0000 7f2e798e8700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e740f5410 0x7f2e74109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:42920/0 (socket says 172.21.15.184:42920) 2024-02-19T22:11:53.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.943+0000 7f2e798e8700 1 -- 172.21.15.184:0/1132799162 learned_addr learned my addr 172.21.15.184:0/1132799162 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:53.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.943+0000 7f2e798e8700 1 -- 172.21.15.184:0/1132799162 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e64005040 con 0x7f2e740f5410 2024-02-19T22:11:53.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.943+0000 7f2e798e8700 1 --2- 172.21.15.184:0/1132799162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e740f5410 0x7f2e74109500 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f2e64005010 tx=0x7f2e6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:53.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.943+0000 7f2e6a7fc700 1 -- 172.21.15.184:0/1132799162 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e64009320 con 0x7f2e740f5410 2024-02-19T22:11:53.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.944+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/1132799162 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e74060420 con 0x7f2e740f5410 2024-02-19T22:11:53.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.944+0000 7f2e6a7fc700 1 -- 172.21.15.184:0/1132799162 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e64013380 con 0x7f2e740f5410 2024-02-19T22:11:53.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.944+0000 7f2e6a7fc700 1 -- 172.21.15.184:0/1132799162 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e64014e50 con 0x7f2e740f5410 2024-02-19T22:11:53.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.944+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/1132799162 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e74109bb0 con 0x7f2e740f5410 2024-02-19T22:11:53.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.944+0000 7f2e6a7fc700 1 -- 172.21.15.184:0/1132799162 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e64014420 con 0x7f2e740f5410 2024-02-19T22:11:53.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.944+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/1132799162 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e74059100 con 0x7f2e740f5410 2024-02-19T22:11:53.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.944+0000 7f2e6a7fc700 1 --2- 172.21.15.184:0/1132799162 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2e6002ff10 0x7f2e600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:53.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.945+0000 7f2e6a7fc700 1 -- 172.21.15.184:0/1132799162 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e64040e30 con 0x7f2e740f5410 2024-02-19T22:11:53.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.945+0000 7f2e790e7700 1 --2- 172.21.15.184:0/1132799162 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2e6002ff10 0x7f2e600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:53.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.945+0000 7f2e790e7700 1 --2- 172.21.15.184:0/1132799162 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2e6002ff10 0x7f2e600323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f2e700096a0 tx=0x7f2e70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:53.950 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:53.949+0000 7f2e6a7fc700 1 -- 172.21.15.184:0/1132799162 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e640095f0 con 0x7f2e740f5410 2024-02-19T22:11:54.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:51.858868+0000 mgr.smithi176.faoogw (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:54.156 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:54.155+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/1132799162 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e74045ad0 con 0x7f2e740f5410 2024-02-19T22:11:54.157 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:54.157+0000 7f2e6a7fc700 1 -- 172.21.15.184:0/1132799162 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2e6401db70 con 0x7f2e740f5410 2024-02-19T22:11:54.157 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:54.157 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:54.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:54.157+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/1132799162 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2e6002ff10 msgr2=0x7f2e600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:54.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:54.157+0000 7f2e7bb4c700 1 --2- 172.21.15.184:0/1132799162 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2e6002ff10 0x7f2e600323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f2e700096a0 tx=0x7f2e70006b40).stop 2024-02-19T22:11:54.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:54.157+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/1132799162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e740f5410 msgr2=0x7f2e74109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:54.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:54.157+0000 7f2e7bb4c700 1 --2- 172.21.15.184:0/1132799162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e740f5410 0x7f2e74109500 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f2e64005010 tx=0x7f2e6400b040).stop 2024-02-19T22:11:54.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:54.157+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/1132799162 shutdown_connections 2024-02-19T22:11:54.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:54.157+0000 7f2e7bb4c700 1 --2- 172.21.15.184:0/1132799162 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2e6002ff10 0x7f2e600323d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:54.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:54.157+0000 7f2e7bb4c700 1 --2- 172.21.15.184:0/1132799162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2e740f5410 0x7f2e74109500 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:54.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:54.157+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/1132799162 >> 172.21.15.184:0/1132799162 conn(0x7f2e740f0b70 msgr2=0x7f2e740f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:54.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:54.157+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/1132799162 shutdown_connections 2024-02-19T22:11:54.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:54.157+0000 7f2e7bb4c700 1 -- 172.21.15.184:0/1132799162 wait complete. 2024-02-19T22:11:54.160 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:55.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:54 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:54.157135+0000 mon.smithi176 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.184:0/1132799162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:55.805 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:55.805 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:11:56.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:55 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:53.859089+0000 mgr.smithi176.faoogw (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:57.593 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.590+0000 7f804f494700 1 -- 172.21.15.184:0/2621144553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80480f5410 msgr2=0x7f80480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:57.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.590+0000 7f804f494700 1 --2- 172.21.15.184:0/2621144553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80480f5410 0x7f80480f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f8044009fb0 tx=0x7f8044009b30).stop 2024-02-19T22:11:57.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.590+0000 7f804f494700 1 -- 172.21.15.184:0/2621144553 shutdown_connections 2024-02-19T22:11:57.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.590+0000 7f804f494700 1 --2- 172.21.15.184:0/2621144553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80480f5410 0x7f80480f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:57.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.590+0000 7f804f494700 1 -- 172.21.15.184:0/2621144553 >> 172.21.15.184:0/2621144553 conn(0x7f80480f0b70 msgr2=0x7f80480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:57.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.590+0000 7f804f494700 1 -- 172.21.15.184:0/2621144553 shutdown_connections 2024-02-19T22:11:57.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.590+0000 7f804f494700 1 -- 172.21.15.184:0/2621144553 wait complete. 2024-02-19T22:11:57.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.591+0000 7f804f494700 1 Processor -- start 2024-02-19T22:11:57.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.591+0000 7f804f494700 1 -- start start 2024-02-19T22:11:57.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.591+0000 7f804f494700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80480f5410 0x7f80480fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:57.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.591+0000 7f804f494700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80480fb960 con 0x7f80480f5410 2024-02-19T22:11:57.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.591+0000 7f804d230700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80480f5410 0x7f80480fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:57.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.591+0000 7f804d230700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80480f5410 0x7f80480fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:59556/0 (socket says 172.21.15.184:59556) 2024-02-19T22:11:57.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.591+0000 7f804d230700 1 -- 172.21.15.184:0/2004178052 learned_addr learned my addr 172.21.15.184:0/2004178052 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:11:57.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.592+0000 7f804d230700 1 -- 172.21.15.184:0/2004178052 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8044005040 con 0x7f80480f5410 2024-02-19T22:11:57.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.592+0000 7f804d230700 1 --2- 172.21.15.184:0/2004178052 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80480f5410 0x7f80480fd4e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f8044005010 tx=0x7f804400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:57.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.592+0000 7f803dffb700 1 -- 172.21.15.184:0/2004178052 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8044009320 con 0x7f80480f5410 2024-02-19T22:11:57.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.592+0000 7f804f494700 1 -- 172.21.15.184:0/2004178052 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80480fbb00 con 0x7f80480f5410 2024-02-19T22:11:57.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.592+0000 7f803dffb700 1 -- 172.21.15.184:0/2004178052 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8044013380 con 0x7f80480f5410 2024-02-19T22:11:57.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.592+0000 7f803dffb700 1 -- 172.21.15.184:0/2004178052 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8044014e50 con 0x7f80480f5410 2024-02-19T22:11:57.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.592+0000 7f804f494700 1 -- 172.21.15.184:0/2004178052 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80480fbf90 con 0x7f80480f5410 2024-02-19T22:11:57.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.593+0000 7f803dffb700 1 -- 172.21.15.184:0/2004178052 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80440263a0 con 0x7f80480f5410 2024-02-19T22:11:57.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.593+0000 7f804f494700 1 -- 172.21.15.184:0/2004178052 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8048059100 con 0x7f80480f5410 2024-02-19T22:11:57.597 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.593+0000 7f803dffb700 1 --2- 172.21.15.184:0/2004178052 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f803402ff10 0x7f80340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:11:57.597 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.593+0000 7f803dffb700 1 -- 172.21.15.184:0/2004178052 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8044041200 con 0x7f80480f5410 2024-02-19T22:11:57.597 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.593+0000 7f804ca2f700 1 --2- 172.21.15.184:0/2004178052 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f803402ff10 0x7f80340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:11:57.597 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.594+0000 7f804ca2f700 1 --2- 172.21.15.184:0/2004178052 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f803402ff10 0x7f80340323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f80380096a0 tx=0x7f8038006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:11:57.599 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.598+0000 7f803dffb700 1 -- 172.21.15.184:0/2004178052 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80440126f0 con 0x7f80480f5410 2024-02-19T22:11:57.805 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.803+0000 7f804f494700 1 -- 172.21.15.184:0/2004178052 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80480fafa0 con 0x7f80480f5410 2024-02-19T22:11:57.805 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.804+0000 7f803dffb700 1 -- 172.21.15.184:0/2004178052 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f80440126f0 con 0x7f80480f5410 2024-02-19T22:11:57.805 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:11:57.806 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:11:57.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.805+0000 7f804f494700 1 -- 172.21.15.184:0/2004178052 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f803402ff10 msgr2=0x7f80340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:57.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.805+0000 7f804f494700 1 --2- 172.21.15.184:0/2004178052 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f803402ff10 0x7f80340323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f80380096a0 tx=0x7f8038006b40).stop 2024-02-19T22:11:57.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.805+0000 7f804f494700 1 -- 172.21.15.184:0/2004178052 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80480f5410 msgr2=0x7f80480fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:11:57.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.805+0000 7f804f494700 1 --2- 172.21.15.184:0/2004178052 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80480f5410 0x7f80480fd4e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f8044005010 tx=0x7f804400b040).stop 2024-02-19T22:11:57.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.806+0000 7f804f494700 1 -- 172.21.15.184:0/2004178052 shutdown_connections 2024-02-19T22:11:57.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.806+0000 7f804f494700 1 --2- 172.21.15.184:0/2004178052 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f803402ff10 0x7f80340323d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:57.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.806+0000 7f804f494700 1 --2- 172.21.15.184:0/2004178052 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f80480f5410 0x7f80480fd4e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:11:57.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.806+0000 7f804f494700 1 -- 172.21.15.184:0/2004178052 >> 172.21.15.184:0/2004178052 conn(0x7f80480f0b70 msgr2=0x7f804805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:11:57.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.806+0000 7f804f494700 1 -- 172.21.15.184:0/2004178052 shutdown_connections 2024-02-19T22:11:57.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:11:57.806+0000 7f804f494700 1 -- 172.21.15.184:0/2004178052 wait complete. 2024-02-19T22:11:57.808 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:11:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:55.859364+0000 mgr.smithi176.faoogw (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:11:59.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:58 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:11:57.804947+0000 mon.smithi176 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.184:0/2004178052' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:11:59.441 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:11:59.441 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:00.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:11:59 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:57.859663+0000 mgr.smithi176.faoogw (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:01.299 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.298+0000 7fd041b84700 1 -- 172.21.15.184:0/4036409552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03c0f51a0 msgr2=0x7fd03c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:01.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.298+0000 7fd041b84700 1 --2- 172.21.15.184:0/4036409552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03c0f51a0 0x7fd03c0f55c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fd030009fb0 tx=0x7fd030009b30).stop 2024-02-19T22:12:01.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.298+0000 7fd041b84700 1 -- 172.21.15.184:0/4036409552 shutdown_connections 2024-02-19T22:12:01.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.298+0000 7fd041b84700 1 --2- 172.21.15.184:0/4036409552 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03c0f51a0 0x7fd03c0f55c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:01.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.298+0000 7fd041b84700 1 -- 172.21.15.184:0/4036409552 >> 172.21.15.184:0/4036409552 conn(0x7fd03c0f09a0 msgr2=0x7fd03c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:01.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.299+0000 7fd041b84700 1 -- 172.21.15.184:0/4036409552 shutdown_connections 2024-02-19T22:12:01.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.299+0000 7fd041b84700 1 -- 172.21.15.184:0/4036409552 wait complete. 2024-02-19T22:12:01.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.299+0000 7fd041b84700 1 Processor -- start 2024-02-19T22:12:01.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.299+0000 7fd041b84700 1 -- start start 2024-02-19T22:12:01.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.299+0000 7fd041b84700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03c0f51a0 0x7fd03c109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:01.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.299+0000 7fd041b84700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c060280 con 0x7fd03c0f51a0 2024-02-19T22:12:01.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.300+0000 7fd040b82700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03c0f51a0 0x7fd03c109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:01.300 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.300+0000 7fd040b82700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03c0f51a0 0x7fd03c109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:59582/0 (socket says 172.21.15.184:59582) 2024-02-19T22:12:01.301 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.300+0000 7fd040b82700 1 -- 172.21.15.184:0/251667972 learned_addr learned my addr 172.21.15.184:0/251667972 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:01.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.300+0000 7fd040b82700 1 -- 172.21.15.184:0/251667972 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd030005040 con 0x7fd03c0f51a0 2024-02-19T22:12:01.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.300+0000 7fd040b82700 1 --2- 172.21.15.184:0/251667972 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03c0f51a0 0x7fd03c109290 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fd0300089f0 tx=0x7fd03000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:01.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.300+0000 7fd0397fa700 1 -- 172.21.15.184:0/251667972 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd030009320 con 0x7fd03c0f51a0 2024-02-19T22:12:01.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.300+0000 7fd041b84700 1 -- 172.21.15.184:0/251667972 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03c060480 con 0x7fd03c0f51a0 2024-02-19T22:12:01.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.300+0000 7fd0397fa700 1 -- 172.21.15.184:0/251667972 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd030013380 con 0x7fd03c0f51a0 2024-02-19T22:12:01.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.300+0000 7fd0397fa700 1 -- 172.21.15.184:0/251667972 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd030014e20 con 0x7fd03c0f51a0 2024-02-19T22:12:01.303 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.300+0000 7fd041b84700 1 -- 172.21.15.184:0/251667972 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03c1099c0 con 0x7fd03c0f51a0 2024-02-19T22:12:01.303 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.301+0000 7fd0397fa700 1 -- 172.21.15.184:0/251667972 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0300263a0 con 0x7fd03c0f51a0 2024-02-19T22:12:01.303 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.301+0000 7fd041b84700 1 -- 172.21.15.184:0/251667972 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd03c045ad0 con 0x7fd03c0f51a0 2024-02-19T22:12:01.303 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.301+0000 7fd0397fa700 1 --2- 172.21.15.184:0/251667972 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd02c02ff10 0x7fd02c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:01.303 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.301+0000 7fd0397fa700 1 -- 172.21.15.184:0/251667972 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0300411d0 con 0x7fd03c0f51a0 2024-02-19T22:12:01.303 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.302+0000 7fd03bfff700 1 --2- 172.21.15.184:0/251667972 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd02c02ff10 0x7fd02c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:01.303 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.302+0000 7fd03bfff700 1 --2- 172.21.15.184:0/251667972 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd02c02ff10 0x7fd02c0323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fd0280096a0 tx=0x7fd028006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:01.306 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.306+0000 7fd0397fa700 1 -- 172.21.15.184:0/251667972 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0300126f0 con 0x7fd03c0f51a0 2024-02-19T22:12:01.512 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.512+0000 7fd041b84700 1 -- 172.21.15.184:0/251667972 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd03c0fad30 con 0x7fd03c0f51a0 2024-02-19T22:12:01.513 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.513+0000 7fd0397fa700 1 -- 172.21.15.184:0/251667972 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd0300126f0 con 0x7fd03c0f51a0 2024-02-19T22:12:01.513 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:01.513 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:01.514 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.514+0000 7fd041b84700 1 -- 172.21.15.184:0/251667972 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd02c02ff10 msgr2=0x7fd02c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:01.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.514+0000 7fd041b84700 1 --2- 172.21.15.184:0/251667972 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd02c02ff10 0x7fd02c0323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fd0280096a0 tx=0x7fd028006b40).stop 2024-02-19T22:12:01.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.514+0000 7fd041b84700 1 -- 172.21.15.184:0/251667972 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03c0f51a0 msgr2=0x7fd03c109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:01.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.514+0000 7fd041b84700 1 --2- 172.21.15.184:0/251667972 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03c0f51a0 0x7fd03c109290 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fd0300089f0 tx=0x7fd03000b040).stop 2024-02-19T22:12:01.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.514+0000 7fd041b84700 1 -- 172.21.15.184:0/251667972 shutdown_connections 2024-02-19T22:12:01.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.514+0000 7fd041b84700 1 --2- 172.21.15.184:0/251667972 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd02c02ff10 0x7fd02c0323d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:01.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.514+0000 7fd041b84700 1 --2- 172.21.15.184:0/251667972 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd03c0f51a0 0x7fd03c109290 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:01.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.514+0000 7fd041b84700 1 -- 172.21.15.184:0/251667972 >> 172.21.15.184:0/251667972 conn(0x7fd03c0f09a0 msgr2=0x7fd03c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:01.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.514+0000 7fd041b84700 1 -- 172.21.15.184:0/251667972 shutdown_connections 2024-02-19T22:12:01.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:01.514+0000 7fd041b84700 1 -- 172.21.15.184:0/251667972 wait complete. 2024-02-19T22:12:01.516 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:02.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:11:59.859897+0000 mgr.smithi176.faoogw (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:02.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:01.512915+0000 mon.smithi176 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.184:0/251667972' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:03.086 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:03.086 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:04.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:01.860203+0000 mgr.smithi176.faoogw (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:04.842 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.839+0000 7f2cadb80700 1 -- 172.21.15.184:0/1323788827 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ca80f7ae0 msgr2=0x7f2ca80f7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:04.842 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.839+0000 7f2cadb80700 1 --2- 172.21.15.184:0/1323788827 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ca80f7ae0 0x7f2ca80f7f00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f2c98009fb0 tx=0x7f2c98009b30).stop 2024-02-19T22:12:04.842 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.839+0000 7f2cadb80700 1 -- 172.21.15.184:0/1323788827 shutdown_connections 2024-02-19T22:12:04.842 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.839+0000 7f2cadb80700 1 --2- 172.21.15.184:0/1323788827 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ca80f7ae0 0x7f2ca80f7f00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:04.842 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.839+0000 7f2cadb80700 1 -- 172.21.15.184:0/1323788827 >> 172.21.15.184:0/1323788827 conn(0x7f2ca80f0260 msgr2=0x7f2ca80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:04.842 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.840+0000 7f2cadb80700 1 -- 172.21.15.184:0/1323788827 shutdown_connections 2024-02-19T22:12:04.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.840+0000 7f2cadb80700 1 -- 172.21.15.184:0/1323788827 wait complete. 2024-02-19T22:12:04.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.840+0000 7f2cadb80700 1 Processor -- start 2024-02-19T22:12:04.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.840+0000 7f2cadb80700 1 -- start start 2024-02-19T22:12:04.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.840+0000 7f2cadb80700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ca80f7ae0 0x7f2ca80f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:04.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.840+0000 7f2cadb80700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca80f5830 con 0x7f2ca80f7ae0 2024-02-19T22:12:04.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.841+0000 7f2ca77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ca80f7ae0 0x7f2ca80f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:04.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.841+0000 7f2ca77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ca80f7ae0 0x7f2ca80f73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:59590/0 (socket says 172.21.15.184:59590) 2024-02-19T22:12:04.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.841+0000 7f2ca77fe700 1 -- 172.21.15.184:0/3043942960 learned_addr learned my addr 172.21.15.184:0/3043942960 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:04.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.841+0000 7f2ca77fe700 1 -- 172.21.15.184:0/3043942960 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c98005040 con 0x7f2ca80f7ae0 2024-02-19T22:12:04.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.841+0000 7f2ca77fe700 1 --2- 172.21.15.184:0/3043942960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ca80f7ae0 0x7f2ca80f73b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f2c98005010 tx=0x7f2c9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:04.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.841+0000 7f2c9ffff700 1 -- 172.21.15.184:0/3043942960 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c98009320 con 0x7f2ca80f7ae0 2024-02-19T22:12:04.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.841+0000 7f2cadb80700 1 -- 172.21.15.184:0/3043942960 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca80f59d0 con 0x7f2ca80f7ae0 2024-02-19T22:12:04.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.841+0000 7f2c9ffff700 1 -- 172.21.15.184:0/3043942960 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c98013380 con 0x7f2ca80f7ae0 2024-02-19T22:12:04.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.841+0000 7f2c9ffff700 1 -- 172.21.15.184:0/3043942960 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c98014e50 con 0x7f2ca80f7ae0 2024-02-19T22:12:04.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.841+0000 7f2cadb80700 1 -- 172.21.15.184:0/3043942960 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca80f5e60 con 0x7f2ca80f7ae0 2024-02-19T22:12:04.844 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.842+0000 7f2c9ffff700 1 -- 172.21.15.184:0/3043942960 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2c980095f0 con 0x7f2ca80f7ae0 2024-02-19T22:12:04.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.842+0000 7f2c9ffff700 1 --2- 172.21.15.184:0/3043942960 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2c9402ff10 0x7f2c940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:04.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.842+0000 7f2cadb80700 1 -- 172.21.15.184:0/3043942960 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca8059100 con 0x7f2ca80f7ae0 2024-02-19T22:12:04.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.842+0000 7f2c9ffff700 1 -- 172.21.15.184:0/3043942960 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c98040fa0 con 0x7f2ca80f7ae0 2024-02-19T22:12:04.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.843+0000 7f2ca6ffd700 1 --2- 172.21.15.184:0/3043942960 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2c9402ff10 0x7f2c940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:04.845 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.843+0000 7f2ca6ffd700 1 --2- 172.21.15.184:0/3043942960 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2c9402ff10 0x7f2c940323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f2c900096a0 tx=0x7f2c90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:04.847 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:04.847+0000 7f2c9ffff700 1 -- 172.21.15.184:0/3043942960 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c98014420 con 0x7f2ca80f7ae0 2024-02-19T22:12:05.053 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:05.053+0000 7f2cadb80700 1 -- 172.21.15.184:0/3043942960 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ca80fc100 con 0x7f2ca80f7ae0 2024-02-19T22:12:05.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:05.054+0000 7f2c9ffff700 1 -- 172.21.15.184:0/3043942960 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2c9802c090 con 0x7f2ca80f7ae0 2024-02-19T22:12:05.054 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:05.054 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:05.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:05.055+0000 7f2cadb80700 1 -- 172.21.15.184:0/3043942960 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2c9402ff10 msgr2=0x7f2c940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:05.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:05.055+0000 7f2cadb80700 1 --2- 172.21.15.184:0/3043942960 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2c9402ff10 0x7f2c940323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f2c900096a0 tx=0x7f2c90006b40).stop 2024-02-19T22:12:05.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:05.055+0000 7f2cadb80700 1 -- 172.21.15.184:0/3043942960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ca80f7ae0 msgr2=0x7f2ca80f73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:05.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:05.055+0000 7f2cadb80700 1 --2- 172.21.15.184:0/3043942960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ca80f7ae0 0x7f2ca80f73b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f2c98005010 tx=0x7f2c9800b040).stop 2024-02-19T22:12:05.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:05.055+0000 7f2cadb80700 1 -- 172.21.15.184:0/3043942960 shutdown_connections 2024-02-19T22:12:05.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:05.055+0000 7f2cadb80700 1 --2- 172.21.15.184:0/3043942960 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2c9402ff10 0x7f2c940323d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:05.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:05.055+0000 7f2cadb80700 1 --2- 172.21.15.184:0/3043942960 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2ca80f7ae0 0x7f2ca80f73b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:05.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:05.055+0000 7f2cadb80700 1 -- 172.21.15.184:0/3043942960 >> 172.21.15.184:0/3043942960 conn(0x7f2ca80f0260 msgr2=0x7f2ca80f0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:05.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:05.056+0000 7f2cadb80700 1 -- 172.21.15.184:0/3043942960 shutdown_connections 2024-02-19T22:12:05.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:05.056+0000 7f2cadb80700 1 -- 172.21.15.184:0/3043942960 wait complete. 2024-02-19T22:12:05.057 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:06.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:03.860546+0000 mgr.smithi176.faoogw (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:06.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:05.054106+0000 mon.smithi176 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.184:0/3043942960' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:06.714 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:06.714 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:08.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:05.860874+0000 mgr.smithi176.faoogw (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:08.548 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.546+0000 7fe707cd2700 1 -- 172.21.15.184:0/2819513977 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000f6510 msgr2=0x7fe7000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:08.549 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.546+0000 7fe707cd2700 1 --2- 172.21.15.184:0/2819513977 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000f6510 0x7fe7000f6930 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fe6f0009fb0 tx=0x7fe6f0009b30).stop 2024-02-19T22:12:08.549 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.546+0000 7fe707cd2700 1 -- 172.21.15.184:0/2819513977 shutdown_connections 2024-02-19T22:12:08.549 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.546+0000 7fe707cd2700 1 --2- 172.21.15.184:0/2819513977 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000f6510 0x7fe7000f6930 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:08.549 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.546+0000 7fe707cd2700 1 -- 172.21.15.184:0/2819513977 >> 172.21.15.184:0/2819513977 conn(0x7fe7000f1c70 msgr2=0x7fe7000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:08.549 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.547+0000 7fe707cd2700 1 -- 172.21.15.184:0/2819513977 shutdown_connections 2024-02-19T22:12:08.549 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.547+0000 7fe707cd2700 1 -- 172.21.15.184:0/2819513977 wait complete. 2024-02-19T22:12:08.549 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.547+0000 7fe707cd2700 1 Processor -- start 2024-02-19T22:12:08.549 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.547+0000 7fe707cd2700 1 -- start start 2024-02-19T22:12:08.549 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.547+0000 7fe707cd2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000f6510 0x7fe7001067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:08.549 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.547+0000 7fe707cd2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe700106cd0 con 0x7fe7000f6510 2024-02-19T22:12:08.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.548+0000 7fe705a6e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000f6510 0x7fe7001067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:08.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.548+0000 7fe705a6e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000f6510 0x7fe7001067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:45456/0 (socket says 172.21.15.184:45456) 2024-02-19T22:12:08.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.548+0000 7fe705a6e700 1 -- 172.21.15.184:0/1627080804 learned_addr learned my addr 172.21.15.184:0/1627080804 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:08.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.548+0000 7fe705a6e700 1 -- 172.21.15.184:0/1627080804 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f0005040 con 0x7fe7000f6510 2024-02-19T22:12:08.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.548+0000 7fe705a6e700 1 --2- 172.21.15.184:0/1627080804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000f6510 0x7fe7001067e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fe6f0005010 tx=0x7fe6f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:08.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.548+0000 7fe6f67fc700 1 -- 172.21.15.184:0/1627080804 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6f0009320 con 0x7fe7000f6510 2024-02-19T22:12:08.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.548+0000 7fe707cd2700 1 -- 172.21.15.184:0/1627080804 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe700106ed0 con 0x7fe7000f6510 2024-02-19T22:12:08.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.548+0000 7fe6f67fc700 1 -- 172.21.15.184:0/1627080804 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6f0013380 con 0x7fe7000f6510 2024-02-19T22:12:08.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.548+0000 7fe6f67fc700 1 -- 172.21.15.184:0/1627080804 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6f0014e50 con 0x7fe7000f6510 2024-02-19T22:12:08.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.548+0000 7fe707cd2700 1 -- 172.21.15.184:0/1627080804 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe700107360 con 0x7fe7000f6510 2024-02-19T22:12:08.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.549+0000 7fe6f67fc700 1 -- 172.21.15.184:0/1627080804 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe6f00095f0 con 0x7fe7000f6510 2024-02-19T22:12:08.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.549+0000 7fe707cd2700 1 -- 172.21.15.184:0/1627080804 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe700045ad0 con 0x7fe7000f6510 2024-02-19T22:12:08.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.549+0000 7fe6f67fc700 1 --2- 172.21.15.184:0/1627080804 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe6ec02ff10 0x7fe6ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:08.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.549+0000 7fe6f67fc700 1 -- 172.21.15.184:0/1627080804 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe6f0040e90 con 0x7fe7000f6510 2024-02-19T22:12:08.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.550+0000 7fe70526d700 1 --2- 172.21.15.184:0/1627080804 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe6ec02ff10 0x7fe6ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:08.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.550+0000 7fe70526d700 1 --2- 172.21.15.184:0/1627080804 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe6ec02ff10 0x7fe6ec0323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe6fc0096a0 tx=0x7fe6fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:08.555 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.554+0000 7fe6f67fc700 1 -- 172.21.15.184:0/1627080804 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe6f0014420 con 0x7fe7000f6510 2024-02-19T22:12:08.766 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.764+0000 7fe707cd2700 1 -- 172.21.15.184:0/1627080804 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe700108210 con 0x7fe7000f6510 2024-02-19T22:12:08.766 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.765+0000 7fe6f67fc700 1 -- 172.21.15.184:0/1627080804 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe6f0010230 con 0x7fe7000f6510 2024-02-19T22:12:08.766 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:08.767 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:08.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.766+0000 7fe707cd2700 1 -- 172.21.15.184:0/1627080804 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe6ec02ff10 msgr2=0x7fe6ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:08.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.766+0000 7fe707cd2700 1 --2- 172.21.15.184:0/1627080804 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe6ec02ff10 0x7fe6ec0323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe6fc0096a0 tx=0x7fe6fc006b40).stop 2024-02-19T22:12:08.767 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.766+0000 7fe707cd2700 1 -- 172.21.15.184:0/1627080804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000f6510 msgr2=0x7fe7001067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:08.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.766+0000 7fe707cd2700 1 --2- 172.21.15.184:0/1627080804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000f6510 0x7fe7001067e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fe6f0005010 tx=0x7fe6f000b040).stop 2024-02-19T22:12:08.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.766+0000 7fe707cd2700 1 -- 172.21.15.184:0/1627080804 shutdown_connections 2024-02-19T22:12:08.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.766+0000 7fe707cd2700 1 --2- 172.21.15.184:0/1627080804 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe6ec02ff10 0x7fe6ec0323d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:08.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.766+0000 7fe707cd2700 1 --2- 172.21.15.184:0/1627080804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe7000f6510 0x7fe7001067e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:08.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.766+0000 7fe707cd2700 1 -- 172.21.15.184:0/1627080804 >> 172.21.15.184:0/1627080804 conn(0x7fe7000f1c70 msgr2=0x7fe7000f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:08.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.766+0000 7fe707cd2700 1 -- 172.21.15.184:0/1627080804 shutdown_connections 2024-02-19T22:12:08.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:08.766+0000 7fe707cd2700 1 -- 172.21.15.184:0/1627080804 wait complete. 2024-02-19T22:12:08.768 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:09.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:08 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:08.765916+0000 mon.smithi176 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.184:0/1627080804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:10.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:07.861136+0000 mgr.smithi176.faoogw (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:10.419 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:10.420 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:12.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:09.861401+0000 mgr.smithi176.faoogw (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:12.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.251+0000 7fcc65925700 1 -- 172.21.15.184:0/1553242165 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc600f6510 msgr2=0x7fcc600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:12.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.251+0000 7fcc65925700 1 --2- 172.21.15.184:0/1553242165 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc600f6510 0x7fcc600f6930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fcc54009fb0 tx=0x7fcc54009b30).stop 2024-02-19T22:12:12.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.251+0000 7fcc65925700 1 -- 172.21.15.184:0/1553242165 shutdown_connections 2024-02-19T22:12:12.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.251+0000 7fcc65925700 1 --2- 172.21.15.184:0/1553242165 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc600f6510 0x7fcc600f6930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:12.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.251+0000 7fcc65925700 1 -- 172.21.15.184:0/1553242165 >> 172.21.15.184:0/1553242165 conn(0x7fcc600f1c70 msgr2=0x7fcc600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:12.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.251+0000 7fcc65925700 1 -- 172.21.15.184:0/1553242165 shutdown_connections 2024-02-19T22:12:12.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.251+0000 7fcc65925700 1 -- 172.21.15.184:0/1553242165 wait complete. 2024-02-19T22:12:12.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.252+0000 7fcc65925700 1 Processor -- start 2024-02-19T22:12:12.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.252+0000 7fcc65925700 1 -- start start 2024-02-19T22:12:12.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.252+0000 7fcc65925700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc600f6510 0x7fcc601067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:12.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.252+0000 7fcc65925700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc60106cd0 con 0x7fcc600f6510 2024-02-19T22:12:12.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.252+0000 7fcc5effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc600f6510 0x7fcc601067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:12.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.252+0000 7fcc5effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc600f6510 0x7fcc601067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:45468/0 (socket says 172.21.15.184:45468) 2024-02-19T22:12:12.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.252+0000 7fcc5effd700 1 -- 172.21.15.184:0/4194597033 learned_addr learned my addr 172.21.15.184:0/4194597033 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:12.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.253+0000 7fcc5effd700 1 -- 172.21.15.184:0/4194597033 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc54005040 con 0x7fcc600f6510 2024-02-19T22:12:12.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.253+0000 7fcc5effd700 1 --2- 172.21.15.184:0/4194597033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc600f6510 0x7fcc601067e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fcc54005010 tx=0x7fcc5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:12.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.253+0000 7fcc477fe700 1 -- 172.21.15.184:0/4194597033 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc54009320 con 0x7fcc600f6510 2024-02-19T22:12:12.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.253+0000 7fcc65925700 1 -- 172.21.15.184:0/4194597033 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc60106ed0 con 0x7fcc600f6510 2024-02-19T22:12:12.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.253+0000 7fcc477fe700 1 -- 172.21.15.184:0/4194597033 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc54013380 con 0x7fcc600f6510 2024-02-19T22:12:12.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.253+0000 7fcc477fe700 1 -- 172.21.15.184:0/4194597033 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc54014e50 con 0x7fcc600f6510 2024-02-19T22:12:12.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.253+0000 7fcc65925700 1 -- 172.21.15.184:0/4194597033 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc60107360 con 0x7fcc600f6510 2024-02-19T22:12:12.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.254+0000 7fcc477fe700 1 -- 172.21.15.184:0/4194597033 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc54014420 con 0x7fcc600f6510 2024-02-19T22:12:12.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.254+0000 7fcc65925700 1 -- 172.21.15.184:0/4194597033 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc60045ad0 con 0x7fcc600f6510 2024-02-19T22:12:12.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.254+0000 7fcc477fe700 1 --2- 172.21.15.184:0/4194597033 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcc4802ff10 0x7fcc480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:12.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.254+0000 7fcc477fe700 1 -- 172.21.15.184:0/4194597033 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc54040e30 con 0x7fcc600f6510 2024-02-19T22:12:12.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.254+0000 7fcc5e7fc700 1 --2- 172.21.15.184:0/4194597033 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcc4802ff10 0x7fcc480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:12.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.255+0000 7fcc5e7fc700 1 --2- 172.21.15.184:0/4194597033 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcc4802ff10 0x7fcc480323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fcc500096a0 tx=0x7fcc50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:12.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.259+0000 7fcc477fe700 1 -- 172.21.15.184:0/4194597033 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc540095f0 con 0x7fcc600f6510 2024-02-19T22:12:12.465 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.464+0000 7fcc65925700 1 -- 172.21.15.184:0/4194597033 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc60108210 con 0x7fcc600f6510 2024-02-19T22:12:12.466 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.465+0000 7fcc477fe700 1 -- 172.21.15.184:0/4194597033 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcc5401db70 con 0x7fcc600f6510 2024-02-19T22:12:12.466 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:12.466 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:12.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.467+0000 7fcc65925700 1 -- 172.21.15.184:0/4194597033 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcc4802ff10 msgr2=0x7fcc480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:12.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.467+0000 7fcc65925700 1 --2- 172.21.15.184:0/4194597033 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcc4802ff10 0x7fcc480323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fcc500096a0 tx=0x7fcc50006b40).stop 2024-02-19T22:12:12.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.467+0000 7fcc65925700 1 -- 172.21.15.184:0/4194597033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc600f6510 msgr2=0x7fcc601067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:12.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.467+0000 7fcc65925700 1 --2- 172.21.15.184:0/4194597033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc600f6510 0x7fcc601067e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fcc54005010 tx=0x7fcc5400b040).stop 2024-02-19T22:12:12.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.467+0000 7fcc65925700 1 -- 172.21.15.184:0/4194597033 shutdown_connections 2024-02-19T22:12:12.469 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.467+0000 7fcc65925700 1 --2- 172.21.15.184:0/4194597033 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcc4802ff10 0x7fcc480323d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:12.469 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.467+0000 7fcc65925700 1 --2- 172.21.15.184:0/4194597033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc600f6510 0x7fcc601067e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:12.469 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.467+0000 7fcc65925700 1 -- 172.21.15.184:0/4194597033 >> 172.21.15.184:0/4194597033 conn(0x7fcc600f1c70 msgr2=0x7fcc600f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:12.469 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.467+0000 7fcc65925700 1 -- 172.21.15.184:0/4194597033 shutdown_connections 2024-02-19T22:12:12.469 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:12.467+0000 7fcc65925700 1 -- 172.21.15.184:0/4194597033 wait complete. 2024-02-19T22:12:12.469 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:13.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:12 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:12.466376+0000 mon.smithi176 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.184:0/4194597033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:14.116 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:14.116 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:14.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:13 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:11.861701+0000 mgr.smithi176.faoogw (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:15.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.890+0000 7f94bf5c4700 1 -- 172.21.15.184:0/3616482287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b80f5410 msgr2=0x7f94b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:15.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.890+0000 7f94bf5c4700 1 --2- 172.21.15.184:0/3616482287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b80f5410 0x7f94b80f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f94a8009fb0 tx=0x7f94a8009b30).stop 2024-02-19T22:12:15.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.890+0000 7f94bf5c4700 1 -- 172.21.15.184:0/3616482287 shutdown_connections 2024-02-19T22:12:15.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.890+0000 7f94bf5c4700 1 --2- 172.21.15.184:0/3616482287 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b80f5410 0x7f94b80f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:15.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.890+0000 7f94bf5c4700 1 -- 172.21.15.184:0/3616482287 >> 172.21.15.184:0/3616482287 conn(0x7f94b80f0b70 msgr2=0x7f94b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:15.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.890+0000 7f94bf5c4700 1 -- 172.21.15.184:0/3616482287 shutdown_connections 2024-02-19T22:12:15.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.890+0000 7f94bf5c4700 1 -- 172.21.15.184:0/3616482287 wait complete. 2024-02-19T22:12:15.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.891+0000 7f94bf5c4700 1 Processor -- start 2024-02-19T22:12:15.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.891+0000 7f94bf5c4700 1 -- start start 2024-02-19T22:12:15.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.891+0000 7f94bf5c4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b80f5410 0x7f94b81067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:15.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.891+0000 7f94bf5c4700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b8106cc0 con 0x7f94b80f5410 2024-02-19T22:12:15.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.891+0000 7f94bd360700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b80f5410 0x7f94b81067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:15.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.891+0000 7f94bd360700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b80f5410 0x7f94b81067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:45472/0 (socket says 172.21.15.184:45472) 2024-02-19T22:12:15.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.891+0000 7f94bd360700 1 -- 172.21.15.184:0/1366708102 learned_addr learned my addr 172.21.15.184:0/1366708102 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:15.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.891+0000 7f94bd360700 1 -- 172.21.15.184:0/1366708102 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a8005040 con 0x7f94b80f5410 2024-02-19T22:12:15.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.891+0000 7f94bd360700 1 --2- 172.21.15.184:0/1366708102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b80f5410 0x7f94b81067d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f94a8005010 tx=0x7f94a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:15.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.892+0000 7f94adffb700 1 -- 172.21.15.184:0/1366708102 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94a8009320 con 0x7f94b80f5410 2024-02-19T22:12:15.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.892+0000 7f94bf5c4700 1 -- 172.21.15.184:0/1366708102 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b8106ec0 con 0x7f94b80f5410 2024-02-19T22:12:15.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.892+0000 7f94adffb700 1 -- 172.21.15.184:0/1366708102 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94a8013380 con 0x7f94b80f5410 2024-02-19T22:12:15.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.892+0000 7f94adffb700 1 -- 172.21.15.184:0/1366708102 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94a8014e50 con 0x7f94b80f5410 2024-02-19T22:12:15.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.892+0000 7f94bf5c4700 1 -- 172.21.15.184:0/1366708102 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b8107350 con 0x7f94b80f5410 2024-02-19T22:12:15.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.893+0000 7f94bf5c4700 1 -- 172.21.15.184:0/1366708102 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94b8045ad0 con 0x7f94b80f5410 2024-02-19T22:12:15.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.893+0000 7f94adffb700 1 -- 172.21.15.184:0/1366708102 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94a80095f0 con 0x7f94b80f5410 2024-02-19T22:12:15.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.893+0000 7f94adffb700 1 --2- 172.21.15.184:0/1366708102 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f94a402ff10 0x7f94a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:15.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.893+0000 7f94adffb700 1 -- 172.21.15.184:0/1366708102 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94a8040e70 con 0x7f94b80f5410 2024-02-19T22:12:15.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.893+0000 7f94bcb5f700 1 --2- 172.21.15.184:0/1366708102 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f94a402ff10 0x7f94a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:15.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.894+0000 7f94bcb5f700 1 --2- 172.21.15.184:0/1366708102 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f94a402ff10 0x7f94a40323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f94b40096a0 tx=0x7f94b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:15.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:15.898+0000 7f94adffb700 1 -- 172.21.15.184:0/1366708102 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94a8014420 con 0x7f94b80f5410 2024-02-19T22:12:16.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:16.102+0000 7f94bf5c4700 1 -- 172.21.15.184:0/1366708102 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94b8108180 con 0x7f94b80f5410 2024-02-19T22:12:16.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:16.103+0000 7f94adffb700 1 -- 172.21.15.184:0/1366708102 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f94a8010230 con 0x7f94b80f5410 2024-02-19T22:12:16.104 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:16.105 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:16.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:16.105+0000 7f94bf5c4700 1 -- 172.21.15.184:0/1366708102 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f94a402ff10 msgr2=0x7f94a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:16.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:16.105+0000 7f94bf5c4700 1 --2- 172.21.15.184:0/1366708102 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f94a402ff10 0x7f94a40323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f94b40096a0 tx=0x7f94b4006b40).stop 2024-02-19T22:12:16.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:16.105+0000 7f94bf5c4700 1 -- 172.21.15.184:0/1366708102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b80f5410 msgr2=0x7f94b81067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:16.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:16.105+0000 7f94bf5c4700 1 --2- 172.21.15.184:0/1366708102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b80f5410 0x7f94b81067d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f94a8005010 tx=0x7f94a800b040).stop 2024-02-19T22:12:16.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:16.105+0000 7f94bf5c4700 1 -- 172.21.15.184:0/1366708102 shutdown_connections 2024-02-19T22:12:16.107 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:16.105+0000 7f94bf5c4700 1 --2- 172.21.15.184:0/1366708102 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f94a402ff10 0x7f94a40323d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:16.107 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:16.105+0000 7f94bf5c4700 1 --2- 172.21.15.184:0/1366708102 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f94b80f5410 0x7f94b81067d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:16.107 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:16.105+0000 7f94bf5c4700 1 -- 172.21.15.184:0/1366708102 >> 172.21.15.184:0/1366708102 conn(0x7f94b80f0b70 msgr2=0x7f94b805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:16.107 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:16.105+0000 7f94bf5c4700 1 -- 172.21.15.184:0/1366708102 shutdown_connections 2024-02-19T22:12:16.107 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:16.105+0000 7f94bf5c4700 1 -- 172.21.15.184:0/1366708102 wait complete. 2024-02-19T22:12:16.107 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:16.134 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:13.862055+0000 mgr.smithi176.faoogw (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:17.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:16 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:16.104350+0000 mon.smithi176 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.184:0/1366708102' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:17.893 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:17.894 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:18.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:17 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:15.862421+0000 mgr.smithi176.faoogw (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:19.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.682+0000 7f463f59e700 1 -- 172.21.15.184:0/1978189260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46400f51a0 msgr2=0x7f46400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:19.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.682+0000 7f463f59e700 1 --2- 172.21.15.184:0/1978189260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46400f51a0 0x7f46400f55c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f4630009fb0 tx=0x7f4630009b30).stop 2024-02-19T22:12:19.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.682+0000 7f463f59e700 1 -- 172.21.15.184:0/1978189260 shutdown_connections 2024-02-19T22:12:19.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.682+0000 7f463f59e700 1 --2- 172.21.15.184:0/1978189260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46400f51a0 0x7f46400f55c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:19.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.682+0000 7f463f59e700 1 -- 172.21.15.184:0/1978189260 >> 172.21.15.184:0/1978189260 conn(0x7f46400f09a0 msgr2=0x7f46400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:19.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.682+0000 7f463f59e700 1 -- 172.21.15.184:0/1978189260 shutdown_connections 2024-02-19T22:12:19.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.682+0000 7f463f59e700 1 -- 172.21.15.184:0/1978189260 wait complete. 2024-02-19T22:12:19.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.682+0000 7f463f59e700 1 Processor -- start 2024-02-19T22:12:19.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.682+0000 7f463f59e700 1 -- start start 2024-02-19T22:12:19.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.683+0000 7f463f59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46400f51a0 0x7f46401065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:19.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.683+0000 7f463f59e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4640106ad0 con 0x7f46400f51a0 2024-02-19T22:12:19.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.683+0000 7f463e59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46400f51a0 0x7f46401065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:19.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.683+0000 7f463e59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46400f51a0 0x7f46401065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:54454/0 (socket says 172.21.15.184:54454) 2024-02-19T22:12:19.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.683+0000 7f463e59c700 1 -- 172.21.15.184:0/3455303630 learned_addr learned my addr 172.21.15.184:0/3455303630 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:19.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.683+0000 7f463e59c700 1 -- 172.21.15.184:0/3455303630 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4630005040 con 0x7f46400f51a0 2024-02-19T22:12:19.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.683+0000 7f463e59c700 1 --2- 172.21.15.184:0/3455303630 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46400f51a0 0x7f46401065e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f46300089f0 tx=0x7f463000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:19.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.683+0000 7f4636ffd700 1 -- 172.21.15.184:0/3455303630 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4630009320 con 0x7f46400f51a0 2024-02-19T22:12:19.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.683+0000 7f4636ffd700 1 -- 172.21.15.184:0/3455303630 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4630013380 con 0x7f46400f51a0 2024-02-19T22:12:19.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.683+0000 7f463f59e700 1 -- 172.21.15.184:0/3455303630 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4640106cd0 con 0x7f46400f51a0 2024-02-19T22:12:19.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.683+0000 7f4636ffd700 1 -- 172.21.15.184:0/3455303630 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4630014e20 con 0x7f46400f51a0 2024-02-19T22:12:19.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.683+0000 7f463f59e700 1 -- 172.21.15.184:0/3455303630 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4640107160 con 0x7f46400f51a0 2024-02-19T22:12:19.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.684+0000 7f4636ffd700 1 -- 172.21.15.184:0/3455303630 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46300095f0 con 0x7f46400f51a0 2024-02-19T22:12:19.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.684+0000 7f463f59e700 1 -- 172.21.15.184:0/3455303630 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4640045ad0 con 0x7f46400f51a0 2024-02-19T22:12:19.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.684+0000 7f4636ffd700 1 --2- 172.21.15.184:0/3455303630 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f462c02ff60 0x7f462c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:19.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.684+0000 7f4636ffd700 1 -- 172.21.15.184:0/3455303630 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4630040e40 con 0x7f46400f51a0 2024-02-19T22:12:19.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.685+0000 7f463dd9b700 1 --2- 172.21.15.184:0/3455303630 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f462c02ff60 0x7f462c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:19.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.685+0000 7f463dd9b700 1 --2- 172.21.15.184:0/3455303630 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f462c02ff60 0x7f462c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f46280096a0 tx=0x7f4628006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:19.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.689+0000 7f4636ffd700 1 -- 172.21.15.184:0/3455303630 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4630014420 con 0x7f46400f51a0 2024-02-19T22:12:19.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.895+0000 7f463f59e700 1 -- 172.21.15.184:0/3455303630 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4640108010 con 0x7f46400f51a0 2024-02-19T22:12:19.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.896+0000 7f4636ffd700 1 -- 172.21.15.184:0/3455303630 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f463001d980 con 0x7f46400f51a0 2024-02-19T22:12:19.897 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:19.897 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:19.899 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.897+0000 7f463f59e700 1 -- 172.21.15.184:0/3455303630 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f462c02ff60 msgr2=0x7f462c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:19.899 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.897+0000 7f463f59e700 1 --2- 172.21.15.184:0/3455303630 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f462c02ff60 0x7f462c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f46280096a0 tx=0x7f4628006b40).stop 2024-02-19T22:12:19.899 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.897+0000 7f463f59e700 1 -- 172.21.15.184:0/3455303630 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46400f51a0 msgr2=0x7f46401065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:19.899 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.897+0000 7f463f59e700 1 --2- 172.21.15.184:0/3455303630 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46400f51a0 0x7f46401065e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f46300089f0 tx=0x7f463000b040).stop 2024-02-19T22:12:19.899 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.897+0000 7f463f59e700 1 -- 172.21.15.184:0/3455303630 shutdown_connections 2024-02-19T22:12:19.899 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.897+0000 7f463f59e700 1 --2- 172.21.15.184:0/3455303630 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f462c02ff60 0x7f462c032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:19.899 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.897+0000 7f463f59e700 1 --2- 172.21.15.184:0/3455303630 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46400f51a0 0x7f46401065e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:19.899 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.897+0000 7f463f59e700 1 -- 172.21.15.184:0/3455303630 >> 172.21.15.184:0/3455303630 conn(0x7f46400f09a0 msgr2=0x7f46400f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:19.899 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.897+0000 7f463f59e700 1 -- 172.21.15.184:0/3455303630 shutdown_connections 2024-02-19T22:12:19.899 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:19.898+0000 7f463f59e700 1 -- 172.21.15.184:0/3455303630 wait complete. 2024-02-19T22:12:19.900 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:20.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:17.862709+0000 mgr.smithi176.faoogw (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:21.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:20 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:19.897261+0000 mon.smithi176 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.184:0/3455303630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:21.579 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:21.579 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:22.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:19.862934+0000 mgr.smithi176.faoogw (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:23.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:22 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:21.863296+0000 mgr.smithi176.faoogw (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:23.301 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.300+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3401609510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c980f5410 msgr2=0x7f7c980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:23.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.300+0000 7f7c9d84f700 1 --2- 172.21.15.184:0/3401609510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c980f5410 0x7f7c980f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f7c88009fb0 tx=0x7f7c88009b30).stop 2024-02-19T22:12:23.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.300+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3401609510 shutdown_connections 2024-02-19T22:12:23.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.300+0000 7f7c9d84f700 1 --2- 172.21.15.184:0/3401609510 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c980f5410 0x7f7c980f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:23.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.300+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3401609510 >> 172.21.15.184:0/3401609510 conn(0x7f7c980f0b70 msgr2=0x7f7c980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:23.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.300+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3401609510 shutdown_connections 2024-02-19T22:12:23.302 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.300+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3401609510 wait complete. 2024-02-19T22:12:23.304 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.300+0000 7f7c9d84f700 1 Processor -- start 2024-02-19T22:12:23.304 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.301+0000 7f7c9d84f700 1 -- start start 2024-02-19T22:12:23.304 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.301+0000 7f7c9d84f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c980f5410 0x7f7c98105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:23.304 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.301+0000 7f7c9d84f700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c98105c10 con 0x7f7c980f5410 2024-02-19T22:12:23.304 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.301+0000 7f7c96ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c980f5410 0x7f7c98105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:23.304 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.301+0000 7f7c96ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c980f5410 0x7f7c98105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:54466/0 (socket says 172.21.15.184:54466) 2024-02-19T22:12:23.304 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.301+0000 7f7c96ffd700 1 -- 172.21.15.184:0/3260788851 learned_addr learned my addr 172.21.15.184:0/3260788851 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:23.305 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.301+0000 7f7c96ffd700 1 -- 172.21.15.184:0/3260788851 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c88005040 con 0x7f7c980f5410 2024-02-19T22:12:23.305 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.301+0000 7f7c96ffd700 1 --2- 172.21.15.184:0/3260788851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c980f5410 0x7f7c98105720 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f7c88005010 tx=0x7f7c8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:23.305 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.302+0000 7f7c7f7fe700 1 -- 172.21.15.184:0/3260788851 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c88009320 con 0x7f7c980f5410 2024-02-19T22:12:23.305 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.302+0000 7f7c7f7fe700 1 -- 172.21.15.184:0/3260788851 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c88013380 con 0x7f7c980f5410 2024-02-19T22:12:23.305 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.302+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3260788851 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c98105e10 con 0x7f7c980f5410 2024-02-19T22:12:23.305 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.302+0000 7f7c7f7fe700 1 -- 172.21.15.184:0/3260788851 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c88014e50 con 0x7f7c980f5410 2024-02-19T22:12:23.305 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.302+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3260788851 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c981062a0 con 0x7f7c980f5410 2024-02-19T22:12:23.305 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.302+0000 7f7c7f7fe700 1 -- 172.21.15.184:0/3260788851 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c88014420 con 0x7f7c980f5410 2024-02-19T22:12:23.305 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.302+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3260788851 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c98045ad0 con 0x7f7c980f5410 2024-02-19T22:12:23.305 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.303+0000 7f7c7f7fe700 1 --2- 172.21.15.184:0/3260788851 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7c8002ff10 0x7f7c800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:23.306 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.303+0000 7f7c7f7fe700 1 -- 172.21.15.184:0/3260788851 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c88040eb0 con 0x7f7c980f5410 2024-02-19T22:12:23.306 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.303+0000 7f7c967fc700 1 --2- 172.21.15.184:0/3260788851 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7c8002ff10 0x7f7c800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:23.306 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.303+0000 7f7c967fc700 1 --2- 172.21.15.184:0/3260788851 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7c8002ff10 0x7f7c800323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f7c8c0096a0 tx=0x7f7c8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:23.308 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.307+0000 7f7c7f7fe700 1 -- 172.21.15.184:0/3260788851 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c8801dd50 con 0x7f7c980f5410 2024-02-19T22:12:23.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.513+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3260788851 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c98107150 con 0x7f7c980f5410 2024-02-19T22:12:23.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.514+0000 7f7c7f7fe700 1 -- 172.21.15.184:0/3260788851 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7c8801db70 con 0x7f7c980f5410 2024-02-19T22:12:23.516 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:23.516 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:23.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.516+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3260788851 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7c8002ff10 msgr2=0x7f7c800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:23.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.516+0000 7f7c9d84f700 1 --2- 172.21.15.184:0/3260788851 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7c8002ff10 0x7f7c800323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f7c8c0096a0 tx=0x7f7c8c006b40).stop 2024-02-19T22:12:23.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.516+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3260788851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c980f5410 msgr2=0x7f7c98105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:23.518 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.516+0000 7f7c9d84f700 1 --2- 172.21.15.184:0/3260788851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c980f5410 0x7f7c98105720 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f7c88005010 tx=0x7f7c8800b040).stop 2024-02-19T22:12:23.518 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.516+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3260788851 shutdown_connections 2024-02-19T22:12:23.518 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.516+0000 7f7c9d84f700 1 --2- 172.21.15.184:0/3260788851 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7c8002ff10 0x7f7c800323d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:23.518 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.516+0000 7f7c9d84f700 1 --2- 172.21.15.184:0/3260788851 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7c980f5410 0x7f7c98105720 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:23.518 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.516+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3260788851 >> 172.21.15.184:0/3260788851 conn(0x7f7c980f0b70 msgr2=0x7f7c980f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:23.518 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.516+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3260788851 shutdown_connections 2024-02-19T22:12:23.518 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:23.516+0000 7f7c9d84f700 1 -- 172.21.15.184:0/3260788851 wait complete. 2024-02-19T22:12:23.518 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:24.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:23.515533+0000 mon.smithi176 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.184:0/3260788851' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:25.117 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:25.118 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:26.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:23.863599+0000 mgr.smithi176.faoogw (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:26.888 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.883+0000 7fca194df700 1 -- 172.21.15.184:0/1947481012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca140f6510 msgr2=0x7fca140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:26.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.883+0000 7fca194df700 1 --2- 172.21.15.184:0/1947481012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca140f6510 0x7fca140f6930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fca08009fb0 tx=0x7fca08009b30).stop 2024-02-19T22:12:26.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.883+0000 7fca194df700 1 -- 172.21.15.184:0/1947481012 shutdown_connections 2024-02-19T22:12:26.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.883+0000 7fca194df700 1 --2- 172.21.15.184:0/1947481012 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca140f6510 0x7fca140f6930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:26.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.883+0000 7fca194df700 1 -- 172.21.15.184:0/1947481012 >> 172.21.15.184:0/1947481012 conn(0x7fca140f1c70 msgr2=0x7fca140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:26.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.883+0000 7fca194df700 1 -- 172.21.15.184:0/1947481012 shutdown_connections 2024-02-19T22:12:26.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.883+0000 7fca194df700 1 -- 172.21.15.184:0/1947481012 wait complete. 2024-02-19T22:12:26.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.884+0000 7fca194df700 1 Processor -- start 2024-02-19T22:12:26.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.884+0000 7fca194df700 1 -- start start 2024-02-19T22:12:26.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.884+0000 7fca194df700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca140f6510 0x7fca141067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:26.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.884+0000 7fca194df700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca14106cd0 con 0x7fca140f6510 2024-02-19T22:12:26.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.884+0000 7fca12ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca140f6510 0x7fca141067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:26.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.884+0000 7fca12ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca140f6510 0x7fca141067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:41610/0 (socket says 172.21.15.184:41610) 2024-02-19T22:12:26.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.884+0000 7fca12ffd700 1 -- 172.21.15.184:0/2066941983 learned_addr learned my addr 172.21.15.184:0/2066941983 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:26.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.885+0000 7fca12ffd700 1 -- 172.21.15.184:0/2066941983 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca08005040 con 0x7fca140f6510 2024-02-19T22:12:26.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.885+0000 7fca12ffd700 1 --2- 172.21.15.184:0/2066941983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca140f6510 0x7fca141067e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fca08005010 tx=0x7fca0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:26.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.885+0000 7fc9fb7fe700 1 -- 172.21.15.184:0/2066941983 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca08009320 con 0x7fca140f6510 2024-02-19T22:12:26.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.885+0000 7fca194df700 1 -- 172.21.15.184:0/2066941983 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca14106ed0 con 0x7fca140f6510 2024-02-19T22:12:26.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.885+0000 7fc9fb7fe700 1 -- 172.21.15.184:0/2066941983 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca08013380 con 0x7fca140f6510 2024-02-19T22:12:26.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.885+0000 7fca194df700 1 -- 172.21.15.184:0/2066941983 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca14107360 con 0x7fca140f6510 2024-02-19T22:12:26.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.885+0000 7fc9fb7fe700 1 -- 172.21.15.184:0/2066941983 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca08014dd0 con 0x7fca140f6510 2024-02-19T22:12:26.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.886+0000 7fc9fb7fe700 1 -- 172.21.15.184:0/2066941983 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca08014420 con 0x7fca140f6510 2024-02-19T22:12:26.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.886+0000 7fca194df700 1 -- 172.21.15.184:0/2066941983 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca14045ad0 con 0x7fca140f6510 2024-02-19T22:12:26.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.886+0000 7fc9fb7fe700 1 --2- 172.21.15.184:0/2066941983 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc9fc02ff60 0x7fc9fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:26.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.886+0000 7fc9fb7fe700 1 -- 172.21.15.184:0/2066941983 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca08040e70 con 0x7fca140f6510 2024-02-19T22:12:26.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.886+0000 7fca127fc700 1 --2- 172.21.15.184:0/2066941983 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc9fc02ff60 0x7fc9fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:26.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.887+0000 7fca127fc700 1 --2- 172.21.15.184:0/2066941983 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc9fc02ff60 0x7fc9fc032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fca040096a0 tx=0x7fca04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:26.892 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:26.891+0000 7fc9fb7fe700 1 -- 172.21.15.184:0/2066941983 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca080095f0 con 0x7fca140f6510 2024-02-19T22:12:27.097 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:27.096+0000 7fca194df700 1 -- 172.21.15.184:0/2066941983 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca14108210 con 0x7fca140f6510 2024-02-19T22:12:27.098 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:27.097+0000 7fc9fb7fe700 1 -- 172.21.15.184:0/2066941983 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fca0802b2d0 con 0x7fca140f6510 2024-02-19T22:12:27.098 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:27.099 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:27.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:27.099+0000 7fca194df700 1 -- 172.21.15.184:0/2066941983 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc9fc02ff60 msgr2=0x7fc9fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:27.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:27.099+0000 7fca194df700 1 --2- 172.21.15.184:0/2066941983 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc9fc02ff60 0x7fc9fc032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fca040096a0 tx=0x7fca04006b40).stop 2024-02-19T22:12:27.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:27.099+0000 7fca194df700 1 -- 172.21.15.184:0/2066941983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca140f6510 msgr2=0x7fca141067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:27.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:27.099+0000 7fca194df700 1 --2- 172.21.15.184:0/2066941983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca140f6510 0x7fca141067e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fca08005010 tx=0x7fca0800b040).stop 2024-02-19T22:12:27.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:27.099+0000 7fca194df700 1 -- 172.21.15.184:0/2066941983 shutdown_connections 2024-02-19T22:12:27.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:27.099+0000 7fca194df700 1 --2- 172.21.15.184:0/2066941983 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc9fc02ff60 0x7fc9fc032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:27.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:27.099+0000 7fca194df700 1 --2- 172.21.15.184:0/2066941983 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fca140f6510 0x7fca141067e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:27.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:27.099+0000 7fca194df700 1 -- 172.21.15.184:0/2066941983 >> 172.21.15.184:0/2066941983 conn(0x7fca140f1c70 msgr2=0x7fca140f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:27.100 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:27.099+0000 7fca194df700 1 -- 172.21.15.184:0/2066941983 shutdown_connections 2024-02-19T22:12:27.101 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:27.099+0000 7fca194df700 1 -- 172.21.15.184:0/2066941983 wait complete. 2024-02-19T22:12:27.101 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:28.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:25.863897+0000 mgr.smithi176.faoogw (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:28.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:27.098279+0000 mon.smithi176 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.184:0/2066941983' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:28.718 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:28.718 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:30.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:29 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:27.864199+0000 mgr.smithi176.faoogw (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:30.504 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.501+0000 7fa61002c700 1 -- 172.21.15.184:0/790835257 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6080f6510 msgr2=0x7fa6080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:30.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.501+0000 7fa61002c700 1 --2- 172.21.15.184:0/790835257 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6080f6510 0x7fa6080f6930 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fa5f8009fb0 tx=0x7fa5f8009b30).stop 2024-02-19T22:12:30.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.501+0000 7fa61002c700 1 -- 172.21.15.184:0/790835257 shutdown_connections 2024-02-19T22:12:30.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.501+0000 7fa61002c700 1 --2- 172.21.15.184:0/790835257 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6080f6510 0x7fa6080f6930 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:30.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.501+0000 7fa61002c700 1 -- 172.21.15.184:0/790835257 >> 172.21.15.184:0/790835257 conn(0x7fa6080f1c70 msgr2=0x7fa6080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:30.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.501+0000 7fa61002c700 1 -- 172.21.15.184:0/790835257 shutdown_connections 2024-02-19T22:12:30.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.501+0000 7fa61002c700 1 -- 172.21.15.184:0/790835257 wait complete. 2024-02-19T22:12:30.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.502+0000 7fa61002c700 1 Processor -- start 2024-02-19T22:12:30.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.502+0000 7fa61002c700 1 -- start start 2024-02-19T22:12:30.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.502+0000 7fa61002c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6080f6510 0x7fa60806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:30.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.502+0000 7fa61002c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa608069e20 con 0x7fa6080f6510 2024-02-19T22:12:30.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.502+0000 7fa60ddc8700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6080f6510 0x7fa60806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:30.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.502+0000 7fa60ddc8700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6080f6510 0x7fa60806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:41630/0 (socket says 172.21.15.184:41630) 2024-02-19T22:12:30.505 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.502+0000 7fa60ddc8700 1 -- 172.21.15.184:0/790765780 learned_addr learned my addr 172.21.15.184:0/790765780 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:30.506 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.503+0000 7fa60ddc8700 1 -- 172.21.15.184:0/790765780 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f8005040 con 0x7fa6080f6510 2024-02-19T22:12:30.506 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.503+0000 7fa60ddc8700 1 --2- 172.21.15.184:0/790765780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6080f6510 0x7fa60806b9a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fa5f8005010 tx=0x7fa5f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:30.506 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.503+0000 7fa5fe7fc700 1 -- 172.21.15.184:0/790765780 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5f8009320 con 0x7fa6080f6510 2024-02-19T22:12:30.506 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.503+0000 7fa61002c700 1 -- 172.21.15.184:0/790765780 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa608069fc0 con 0x7fa6080f6510 2024-02-19T22:12:30.506 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.503+0000 7fa5fe7fc700 1 -- 172.21.15.184:0/790765780 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5f8013380 con 0x7fa6080f6510 2024-02-19T22:12:30.506 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.503+0000 7fa5fe7fc700 1 -- 172.21.15.184:0/790765780 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5f8014e50 con 0x7fa6080f6510 2024-02-19T22:12:30.506 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.503+0000 7fa61002c700 1 -- 172.21.15.184:0/790765780 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60806a450 con 0x7fa6080f6510 2024-02-19T22:12:30.507 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.504+0000 7fa5fe7fc700 1 -- 172.21.15.184:0/790765780 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5f80095f0 con 0x7fa6080f6510 2024-02-19T22:12:30.507 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.504+0000 7fa61002c700 1 -- 172.21.15.184:0/790765780 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa608059100 con 0x7fa6080f6510 2024-02-19T22:12:30.507 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.504+0000 7fa5fe7fc700 1 --2- 172.21.15.184:0/790765780 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa5f402ff10 0x7fa5f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:30.507 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.504+0000 7fa5fe7fc700 1 -- 172.21.15.184:0/790765780 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5f8040f80 con 0x7fa6080f6510 2024-02-19T22:12:30.507 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.504+0000 7fa60d5c7700 1 --2- 172.21.15.184:0/790765780 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa5f402ff10 0x7fa5f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:30.507 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.505+0000 7fa60d5c7700 1 --2- 172.21.15.184:0/790765780 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa5f402ff10 0x7fa5f40323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fa6040096a0 tx=0x7fa604006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:30.510 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.509+0000 7fa5fe7fc700 1 -- 172.21.15.184:0/790765780 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5f8014420 con 0x7fa6080f6510 2024-02-19T22:12:30.715 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.714+0000 7fa61002c700 1 -- 172.21.15.184:0/790765780 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6080fc0a0 con 0x7fa6080f6510 2024-02-19T22:12:30.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.715+0000 7fa5fe7fc700 1 -- 172.21.15.184:0/790765780 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa5f8010230 con 0x7fa6080f6510 2024-02-19T22:12:30.716 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:30.716 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:30.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.716+0000 7fa61002c700 1 -- 172.21.15.184:0/790765780 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa5f402ff10 msgr2=0x7fa5f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:30.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.716+0000 7fa61002c700 1 --2- 172.21.15.184:0/790765780 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa5f402ff10 0x7fa5f40323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fa6040096a0 tx=0x7fa604006b40).stop 2024-02-19T22:12:30.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.716+0000 7fa61002c700 1 -- 172.21.15.184:0/790765780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6080f6510 msgr2=0x7fa60806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:30.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.716+0000 7fa61002c700 1 --2- 172.21.15.184:0/790765780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6080f6510 0x7fa60806b9a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fa5f8005010 tx=0x7fa5f800b040).stop 2024-02-19T22:12:30.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.716+0000 7fa61002c700 1 -- 172.21.15.184:0/790765780 shutdown_connections 2024-02-19T22:12:30.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.716+0000 7fa61002c700 1 --2- 172.21.15.184:0/790765780 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa5f402ff10 0x7fa5f40323d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:30.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.716+0000 7fa61002c700 1 --2- 172.21.15.184:0/790765780 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6080f6510 0x7fa60806b9a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:30.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.716+0000 7fa61002c700 1 -- 172.21.15.184:0/790765780 >> 172.21.15.184:0/790765780 conn(0x7fa6080f1c70 msgr2=0x7fa6080f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:30.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.716+0000 7fa61002c700 1 -- 172.21.15.184:0/790765780 shutdown_connections 2024-02-19T22:12:30.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:30.716+0000 7fa61002c700 1 -- 172.21.15.184:0/790765780 wait complete. 2024-02-19T22:12:30.719 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:31.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:30 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:30.716038+0000 mon.smithi176 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.184:0/790765780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:32.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:29.864580+0000 mgr.smithi176.faoogw (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:32.353 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:32.354 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:34.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:31.864921+0000 mgr.smithi176.faoogw (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:34.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.170+0000 7efe6d331700 1 -- 172.21.15.184:0/3240433117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe680f5410 msgr2=0x7efe680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:34.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.170+0000 7efe6d331700 1 --2- 172.21.15.184:0/3240433117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe680f5410 0x7efe680f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7efe50009fb0 tx=0x7efe50009b30).stop 2024-02-19T22:12:34.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.170+0000 7efe6d331700 1 -- 172.21.15.184:0/3240433117 shutdown_connections 2024-02-19T22:12:34.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.170+0000 7efe6d331700 1 --2- 172.21.15.184:0/3240433117 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe680f5410 0x7efe680f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:34.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.170+0000 7efe6d331700 1 -- 172.21.15.184:0/3240433117 >> 172.21.15.184:0/3240433117 conn(0x7efe680f0b70 msgr2=0x7efe680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:34.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.170+0000 7efe6d331700 1 -- 172.21.15.184:0/3240433117 shutdown_connections 2024-02-19T22:12:34.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.170+0000 7efe6d331700 1 -- 172.21.15.184:0/3240433117 wait complete. 2024-02-19T22:12:34.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.171+0000 7efe6d331700 1 Processor -- start 2024-02-19T22:12:34.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.171+0000 7efe6d331700 1 -- start start 2024-02-19T22:12:34.174 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.171+0000 7efe6d331700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe680f5410 0x7efe68109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:34.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.171+0000 7efe6d331700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe68060220 con 0x7efe680f5410 2024-02-19T22:12:34.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.171+0000 7efe66ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe680f5410 0x7efe68109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:34.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.172+0000 7efe66ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe680f5410 0x7efe68109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:41648/0 (socket says 172.21.15.184:41648) 2024-02-19T22:12:34.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.172+0000 7efe66ffd700 1 -- 172.21.15.184:0/858605881 learned_addr learned my addr 172.21.15.184:0/858605881 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:34.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.172+0000 7efe66ffd700 1 -- 172.21.15.184:0/858605881 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe50005040 con 0x7efe680f5410 2024-02-19T22:12:34.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.172+0000 7efe66ffd700 1 --2- 172.21.15.184:0/858605881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe680f5410 0x7efe68109500 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7efe500089f0 tx=0x7efe5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:34.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.172+0000 7efe5f7fe700 1 -- 172.21.15.184:0/858605881 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe50009320 con 0x7efe680f5410 2024-02-19T22:12:34.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.172+0000 7efe6d331700 1 -- 172.21.15.184:0/858605881 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe68060420 con 0x7efe680f5410 2024-02-19T22:12:34.175 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.172+0000 7efe5f7fe700 1 -- 172.21.15.184:0/858605881 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe50013380 con 0x7efe680f5410 2024-02-19T22:12:34.176 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.172+0000 7efe5f7fe700 1 -- 172.21.15.184:0/858605881 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe5001ed80 con 0x7efe680f5410 2024-02-19T22:12:34.176 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.172+0000 7efe6d331700 1 -- 172.21.15.184:0/858605881 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe68109bb0 con 0x7efe680f5410 2024-02-19T22:12:34.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.173+0000 7efe5f7fe700 1 -- 172.21.15.184:0/858605881 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe5001e420 con 0x7efe680f5410 2024-02-19T22:12:34.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.173+0000 7efe6d331700 1 -- 172.21.15.184:0/858605881 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe68059100 con 0x7efe680f5410 2024-02-19T22:12:34.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.173+0000 7efe5f7fe700 1 --2- 172.21.15.184:0/858605881 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7efe5402ff60 0x7efe54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:34.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.173+0000 7efe5f7fe700 1 -- 172.21.15.184:0/858605881 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe50040e10 con 0x7efe680f5410 2024-02-19T22:12:34.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.173+0000 7efe667fc700 1 --2- 172.21.15.184:0/858605881 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7efe5402ff60 0x7efe54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:34.177 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.174+0000 7efe667fc700 1 --2- 172.21.15.184:0/858605881 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7efe5402ff60 0x7efe54032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7efe580096a0 tx=0x7efe58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:34.179 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.178+0000 7efe5f7fe700 1 -- 172.21.15.184:0/858605881 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe5001c930 con 0x7efe680f5410 2024-02-19T22:12:34.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.382+0000 7efe6d331700 1 -- 172.21.15.184:0/858605881 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe68045ad0 con 0x7efe680f5410 2024-02-19T22:12:34.384 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.383+0000 7efe5f7fe700 1 -- 172.21.15.184:0/858605881 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efe5001cb40 con 0x7efe680f5410 2024-02-19T22:12:34.384 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:34.384 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:34.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.384+0000 7efe6d331700 1 -- 172.21.15.184:0/858605881 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7efe5402ff60 msgr2=0x7efe54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:34.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.384+0000 7efe6d331700 1 --2- 172.21.15.184:0/858605881 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7efe5402ff60 0x7efe54032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7efe580096a0 tx=0x7efe58006b40).stop 2024-02-19T22:12:34.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.384+0000 7efe6d331700 1 -- 172.21.15.184:0/858605881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe680f5410 msgr2=0x7efe68109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:34.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.384+0000 7efe6d331700 1 --2- 172.21.15.184:0/858605881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe680f5410 0x7efe68109500 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7efe500089f0 tx=0x7efe5000b040).stop 2024-02-19T22:12:34.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.384+0000 7efe6d331700 1 -- 172.21.15.184:0/858605881 shutdown_connections 2024-02-19T22:12:34.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.384+0000 7efe6d331700 1 --2- 172.21.15.184:0/858605881 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7efe5402ff60 0x7efe54032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:34.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.384+0000 7efe6d331700 1 --2- 172.21.15.184:0/858605881 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efe680f5410 0x7efe68109500 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:34.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.384+0000 7efe6d331700 1 -- 172.21.15.184:0/858605881 >> 172.21.15.184:0/858605881 conn(0x7efe680f0b70 msgr2=0x7efe680f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:34.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.384+0000 7efe6d331700 1 -- 172.21.15.184:0/858605881 shutdown_connections 2024-02-19T22:12:34.387 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:34.384+0000 7efe6d331700 1 -- 172.21.15.184:0/858605881 wait complete. 2024-02-19T22:12:34.387 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:35.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:34 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:34.384002+0000 mon.smithi176 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.184:0/858605881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:36.052 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:36.052 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:36.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:33.865279+0000 mgr.smithi176.faoogw (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:35.969731+0000 mon.smithi176 (mon.0) 178 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:12:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:35.972290+0000 mon.smithi176 (mon.0) 179 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:12:37.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.929+0000 7fcdb7fff700 1 -- 172.21.15.184:0/2358830300 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdb80f51a0 msgr2=0x7fcdb80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:37.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.929+0000 7fcdb7fff700 1 --2- 172.21.15.184:0/2358830300 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdb80f51a0 0x7fcdb80f55c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fcda8009fb0 tx=0x7fcda8009b30).stop 2024-02-19T22:12:37.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.929+0000 7fcdb7fff700 1 -- 172.21.15.184:0/2358830300 shutdown_connections 2024-02-19T22:12:37.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.929+0000 7fcdb7fff700 1 --2- 172.21.15.184:0/2358830300 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdb80f51a0 0x7fcdb80f55c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:37.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.929+0000 7fcdb7fff700 1 -- 172.21.15.184:0/2358830300 >> 172.21.15.184:0/2358830300 conn(0x7fcdb80f09a0 msgr2=0x7fcdb80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:37.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.930+0000 7fcdb7fff700 1 -- 172.21.15.184:0/2358830300 shutdown_connections 2024-02-19T22:12:37.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.930+0000 7fcdb7fff700 1 -- 172.21.15.184:0/2358830300 wait complete. 2024-02-19T22:12:37.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.930+0000 7fcdb7fff700 1 Processor -- start 2024-02-19T22:12:37.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.930+0000 7fcdb7fff700 1 -- start start 2024-02-19T22:12:37.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.930+0000 7fcdb7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdb80f51a0 0x7fcdb81065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:37.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.930+0000 7fcdb7fff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb8106ad0 con 0x7fcdb80f51a0 2024-02-19T22:12:37.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.931+0000 7fcdb6ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdb80f51a0 0x7fcdb81065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:37.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.931+0000 7fcdb6ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdb80f51a0 0x7fcdb81065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:33418/0 (socket says 172.21.15.184:33418) 2024-02-19T22:12:37.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.931+0000 7fcdb6ffd700 1 -- 172.21.15.184:0/646983041 learned_addr learned my addr 172.21.15.184:0/646983041 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:37.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.931+0000 7fcdb6ffd700 1 -- 172.21.15.184:0/646983041 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda8005040 con 0x7fcdb80f51a0 2024-02-19T22:12:37.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.931+0000 7fcdb6ffd700 1 --2- 172.21.15.184:0/646983041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdb80f51a0 0x7fcdb81065e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fcda80089f0 tx=0x7fcda800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:37.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.931+0000 7fcd9f7fe700 1 -- 172.21.15.184:0/646983041 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcda8009320 con 0x7fcdb80f51a0 2024-02-19T22:12:37.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.931+0000 7fcdb7fff700 1 -- 172.21.15.184:0/646983041 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb8106cd0 con 0x7fcdb80f51a0 2024-02-19T22:12:37.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.931+0000 7fcd9f7fe700 1 -- 172.21.15.184:0/646983041 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcda8013380 con 0x7fcdb80f51a0 2024-02-19T22:12:37.935 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.931+0000 7fcd9f7fe700 1 -- 172.21.15.184:0/646983041 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcda8014e20 con 0x7fcdb80f51a0 2024-02-19T22:12:37.935 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.931+0000 7fcdb7fff700 1 -- 172.21.15.184:0/646983041 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb8107160 con 0x7fcdb80f51a0 2024-02-19T22:12:37.935 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.932+0000 7fcd9f7fe700 1 -- 172.21.15.184:0/646983041 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcda80263a0 con 0x7fcdb80f51a0 2024-02-19T22:12:37.935 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.932+0000 7fcd9f7fe700 1 --2- 172.21.15.184:0/646983041 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcda002ff10 0x7fcda00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:37.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.932+0000 7fcdb7fff700 1 -- 172.21.15.184:0/646983041 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb8045ad0 con 0x7fcdb80f51a0 2024-02-19T22:12:37.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.932+0000 7fcd9f7fe700 1 -- 172.21.15.184:0/646983041 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcda8041270 con 0x7fcdb80f51a0 2024-02-19T22:12:37.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.933+0000 7fcdb67fc700 1 --2- 172.21.15.184:0/646983041 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcda002ff10 0x7fcda00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:37.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.933+0000 7fcdb67fc700 1 --2- 172.21.15.184:0/646983041 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcda002ff10 0x7fcda00323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fcdac0096a0 tx=0x7fcdac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:37.938 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:37.937+0000 7fcd9f7fe700 1 -- 172.21.15.184:0/646983041 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcda801cac0 con 0x7fcdb80f51a0 2024-02-19T22:12:38.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:35.865631+0000 mgr.smithi176.faoogw (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:38.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:38.141+0000 7fcdb7fff700 1 -- 172.21.15.184:0/646983041 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcdb8108010 con 0x7fcdb80f51a0 2024-02-19T22:12:38.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:38.142+0000 7fcd9f7fe700 1 -- 172.21.15.184:0/646983041 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcda801cac0 con 0x7fcdb80f51a0 2024-02-19T22:12:38.143 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:38.144 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:38.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:38.144+0000 7fcdb7fff700 1 -- 172.21.15.184:0/646983041 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcda002ff10 msgr2=0x7fcda00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:38.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:38.144+0000 7fcdb7fff700 1 --2- 172.21.15.184:0/646983041 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcda002ff10 0x7fcda00323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fcdac0096a0 tx=0x7fcdac006b40).stop 2024-02-19T22:12:38.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:38.144+0000 7fcdb7fff700 1 -- 172.21.15.184:0/646983041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdb80f51a0 msgr2=0x7fcdb81065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:38.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:38.144+0000 7fcdb7fff700 1 --2- 172.21.15.184:0/646983041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdb80f51a0 0x7fcdb81065e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fcda80089f0 tx=0x7fcda800b040).stop 2024-02-19T22:12:38.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:38.144+0000 7fcdb7fff700 1 -- 172.21.15.184:0/646983041 shutdown_connections 2024-02-19T22:12:38.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:38.144+0000 7fcdb7fff700 1 --2- 172.21.15.184:0/646983041 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcda002ff10 0x7fcda00323d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:38.146 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:38.144+0000 7fcdb7fff700 1 --2- 172.21.15.184:0/646983041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcdb80f51a0 0x7fcdb81065e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:38.146 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:38.144+0000 7fcdb7fff700 1 -- 172.21.15.184:0/646983041 >> 172.21.15.184:0/646983041 conn(0x7fcdb80f09a0 msgr2=0x7fcdb80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:38.146 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:38.144+0000 7fcdb7fff700 1 -- 172.21.15.184:0/646983041 shutdown_connections 2024-02-19T22:12:38.146 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:38.144+0000 7fcdb7fff700 1 -- 172.21.15.184:0/646983041 wait complete. 2024-02-19T22:12:38.146 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:39.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:38 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:38.143235+0000 mon.smithi176 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.184:0/646983041' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:39.765 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:39.766 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:40.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:37.865862+0000 mgr.smithi176.faoogw (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:41.569 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.566+0000 7f42fbe17700 1 -- 172.21.15.184:0/608153794 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42f40f47c0 msgr2=0x7f42f40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:41.569 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.566+0000 7f42fbe17700 1 --2- 172.21.15.184:0/608153794 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42f40f47c0 0x7f42f40f4be0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f42e4009fb0 tx=0x7f42e4009b30).stop 2024-02-19T22:12:41.570 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.566+0000 7f42fbe17700 1 -- 172.21.15.184:0/608153794 shutdown_connections 2024-02-19T22:12:41.570 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.566+0000 7f42fbe17700 1 --2- 172.21.15.184:0/608153794 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42f40f47c0 0x7f42f40f4be0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:41.570 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.566+0000 7f42fbe17700 1 -- 172.21.15.184:0/608153794 >> 172.21.15.184:0/608153794 conn(0x7f42f40f02e0 msgr2=0x7f42f40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:41.570 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.566+0000 7f42fbe17700 1 -- 172.21.15.184:0/608153794 shutdown_connections 2024-02-19T22:12:41.570 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.566+0000 7f42fbe17700 1 -- 172.21.15.184:0/608153794 wait complete. 2024-02-19T22:12:41.570 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.567+0000 7f42fbe17700 1 Processor -- start 2024-02-19T22:12:41.570 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.567+0000 7f42fbe17700 1 -- start start 2024-02-19T22:12:41.570 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.567+0000 7f42fbe17700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42f40f47c0 0x7f42f41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:41.570 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.567+0000 7f42fbe17700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f4106cc0 con 0x7f42f40f47c0 2024-02-19T22:12:41.570 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.567+0000 7f42f9bb3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42f40f47c0 0x7f42f41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:41.570 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.567+0000 7f42f9bb3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42f40f47c0 0x7f42f41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:33442/0 (socket says 172.21.15.184:33442) 2024-02-19T22:12:41.570 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.567+0000 7f42f9bb3700 1 -- 172.21.15.184:0/1940141197 learned_addr learned my addr 172.21.15.184:0/1940141197 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:41.570 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.568+0000 7f42f9bb3700 1 -- 172.21.15.184:0/1940141197 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e4005040 con 0x7f42f40f47c0 2024-02-19T22:12:41.571 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.568+0000 7f42f9bb3700 1 --2- 172.21.15.184:0/1940141197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42f40f47c0 0x7f42f41067d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f42e4005010 tx=0x7f42e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:41.571 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.568+0000 7f42ea7fc700 1 -- 172.21.15.184:0/1940141197 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42e4009320 con 0x7f42f40f47c0 2024-02-19T22:12:41.571 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.568+0000 7f42fbe17700 1 -- 172.21.15.184:0/1940141197 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f4106ec0 con 0x7f42f40f47c0 2024-02-19T22:12:41.571 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.568+0000 7f42ea7fc700 1 -- 172.21.15.184:0/1940141197 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42e4013380 con 0x7f42f40f47c0 2024-02-19T22:12:41.571 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.568+0000 7f42ea7fc700 1 -- 172.21.15.184:0/1940141197 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42e4014e50 con 0x7f42f40f47c0 2024-02-19T22:12:41.571 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.568+0000 7f42fbe17700 1 -- 172.21.15.184:0/1940141197 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f4107350 con 0x7f42f40f47c0 2024-02-19T22:12:41.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.569+0000 7f42ea7fc700 1 -- 172.21.15.184:0/1940141197 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42e40263a0 con 0x7f42f40f47c0 2024-02-19T22:12:41.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.569+0000 7f42ea7fc700 1 --2- 172.21.15.184:0/1940141197 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f42e002ff10 0x7f42e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:41.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.569+0000 7f42ea7fc700 1 -- 172.21.15.184:0/1940141197 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42e4040fc0 con 0x7f42f40f47c0 2024-02-19T22:12:41.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.569+0000 7f42fbe17700 1 -- 172.21.15.184:0/1940141197 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f4045ad0 con 0x7f42f40f47c0 2024-02-19T22:12:41.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.569+0000 7f42f93b2700 1 --2- 172.21.15.184:0/1940141197 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f42e002ff10 0x7f42e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:41.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.569+0000 7f42f93b2700 1 --2- 172.21.15.184:0/1940141197 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f42e002ff10 0x7f42e00323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f42f00096a0 tx=0x7f42f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:41.575 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.574+0000 7f42ea7fc700 1 -- 172.21.15.184:0/1940141197 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42e4012920 con 0x7f42f40f47c0 2024-02-19T22:12:41.779 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.777+0000 7f42fbe17700 1 -- 172.21.15.184:0/1940141197 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42f40fc180 con 0x7f42f40f47c0 2024-02-19T22:12:41.780 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.780+0000 7f42ea7fc700 1 -- 172.21.15.184:0/1940141197 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f42e4012330 con 0x7f42f40f47c0 2024-02-19T22:12:41.780 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:41.781 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:41.782 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.781+0000 7f42fbe17700 1 -- 172.21.15.184:0/1940141197 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f42e002ff10 msgr2=0x7f42e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:41.782 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.781+0000 7f42fbe17700 1 --2- 172.21.15.184:0/1940141197 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f42e002ff10 0x7f42e00323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f42f00096a0 tx=0x7f42f0006b40).stop 2024-02-19T22:12:41.782 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.781+0000 7f42fbe17700 1 -- 172.21.15.184:0/1940141197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42f40f47c0 msgr2=0x7f42f41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:41.782 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.781+0000 7f42fbe17700 1 --2- 172.21.15.184:0/1940141197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42f40f47c0 0x7f42f41067d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f42e4005010 tx=0x7f42e400b040).stop 2024-02-19T22:12:41.782 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.781+0000 7f42fbe17700 1 -- 172.21.15.184:0/1940141197 shutdown_connections 2024-02-19T22:12:41.782 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.781+0000 7f42fbe17700 1 --2- 172.21.15.184:0/1940141197 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f42e002ff10 0x7f42e00323d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:41.782 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.781+0000 7f42fbe17700 1 --2- 172.21.15.184:0/1940141197 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f42f40f47c0 0x7f42f41067d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:41.782 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.781+0000 7f42fbe17700 1 -- 172.21.15.184:0/1940141197 >> 172.21.15.184:0/1940141197 conn(0x7f42f40f02e0 msgr2=0x7f42f40f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:41.783 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.781+0000 7f42fbe17700 1 -- 172.21.15.184:0/1940141197 shutdown_connections 2024-02-19T22:12:41.783 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:41.781+0000 7f42fbe17700 1 -- 172.21.15.184:0/1940141197 wait complete. 2024-02-19T22:12:41.783 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:42.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:39.866152+0000 mgr.smithi176.faoogw (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:43.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:42 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:41.780337+0000 mon.smithi176 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.184:0/1940141197' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:43.378 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:43.378 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:44.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:41.866413+0000 mgr.smithi176.faoogw (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:45.116 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.115+0000 7f73469fe700 1 -- 172.21.15.184:0/1165294842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73400f6510 msgr2=0x7f73400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:45.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.115+0000 7f73469fe700 1 --2- 172.21.15.184:0/1165294842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73400f6510 0x7f73400f6930 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f7334009fb0 tx=0x7f7334009b30).stop 2024-02-19T22:12:45.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.115+0000 7f73469fe700 1 -- 172.21.15.184:0/1165294842 shutdown_connections 2024-02-19T22:12:45.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.115+0000 7f73469fe700 1 --2- 172.21.15.184:0/1165294842 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73400f6510 0x7f73400f6930 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:45.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.115+0000 7f73469fe700 1 -- 172.21.15.184:0/1165294842 >> 172.21.15.184:0/1165294842 conn(0x7f73400f1c70 msgr2=0x7f73400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:45.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.115+0000 7f73469fe700 1 -- 172.21.15.184:0/1165294842 shutdown_connections 2024-02-19T22:12:45.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.115+0000 7f73469fe700 1 -- 172.21.15.184:0/1165294842 wait complete. 2024-02-19T22:12:45.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.115+0000 7f73469fe700 1 Processor -- start 2024-02-19T22:12:45.117 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.116+0000 7f73469fe700 1 -- start start 2024-02-19T22:12:45.118 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.116+0000 7f73469fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73400f6510 0x7f734006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:45.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.116+0000 7f73469fe700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7340069e20 con 0x7f73400f6510 2024-02-19T22:12:45.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.116+0000 7f733ffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73400f6510 0x7f734006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:45.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.116+0000 7f733ffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73400f6510 0x7f734006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:33460/0 (socket says 172.21.15.184:33460) 2024-02-19T22:12:45.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.116+0000 7f733ffff700 1 -- 172.21.15.184:0/3696027193 learned_addr learned my addr 172.21.15.184:0/3696027193 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:45.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.116+0000 7f733ffff700 1 -- 172.21.15.184:0/3696027193 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7334005040 con 0x7f73400f6510 2024-02-19T22:12:45.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.116+0000 7f733ffff700 1 --2- 172.21.15.184:0/3696027193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73400f6510 0x7f734006b9a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f7334005010 tx=0x7f733400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:45.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.116+0000 7f733cff9700 1 -- 172.21.15.184:0/3696027193 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7334009320 con 0x7f73400f6510 2024-02-19T22:12:45.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.117+0000 7f733cff9700 1 -- 172.21.15.184:0/3696027193 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7334013380 con 0x7f73400f6510 2024-02-19T22:12:45.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.117+0000 7f73469fe700 1 -- 172.21.15.184:0/3696027193 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7340069fc0 con 0x7f73400f6510 2024-02-19T22:12:45.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.117+0000 7f733cff9700 1 -- 172.21.15.184:0/3696027193 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7334014e50 con 0x7f73400f6510 2024-02-19T22:12:45.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.117+0000 7f73469fe700 1 -- 172.21.15.184:0/3696027193 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f734006a450 con 0x7f73400f6510 2024-02-19T22:12:45.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.117+0000 7f733cff9700 1 -- 172.21.15.184:0/3696027193 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f73340095f0 con 0x7f73400f6510 2024-02-19T22:12:45.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.117+0000 7f73469fe700 1 -- 172.21.15.184:0/3696027193 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7340059100 con 0x7f73400f6510 2024-02-19T22:12:45.119 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.117+0000 7f733cff9700 1 --2- 172.21.15.184:0/3696027193 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f732802ff10 0x7f73280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:45.120 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.118+0000 7f733cff9700 1 -- 172.21.15.184:0/3696027193 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7334040fa0 con 0x7f73400f6510 2024-02-19T22:12:45.120 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.118+0000 7f733f7fe700 1 --2- 172.21.15.184:0/3696027193 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f732802ff10 0x7f73280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:45.120 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.118+0000 7f733f7fe700 1 --2- 172.21.15.184:0/3696027193 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f732802ff10 0x7f73280323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f73300096a0 tx=0x7f7330006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:45.123 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.122+0000 7f733cff9700 1 -- 172.21.15.184:0/3696027193 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7334014420 con 0x7f73400f6510 2024-02-19T22:12:45.326 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.325+0000 7f73469fe700 1 -- 172.21.15.184:0/3696027193 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73400fc0a0 con 0x7f73400f6510 2024-02-19T22:12:45.327 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.325+0000 7f733cff9700 1 -- 172.21.15.184:0/3696027193 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7334010230 con 0x7f73400f6510 2024-02-19T22:12:45.327 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:45.327 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:45.328 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.327+0000 7f73469fe700 1 -- 172.21.15.184:0/3696027193 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f732802ff10 msgr2=0x7f73280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:45.328 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.327+0000 7f73469fe700 1 --2- 172.21.15.184:0/3696027193 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f732802ff10 0x7f73280323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f73300096a0 tx=0x7f7330006b40).stop 2024-02-19T22:12:45.328 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.327+0000 7f73469fe700 1 -- 172.21.15.184:0/3696027193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73400f6510 msgr2=0x7f734006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:45.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.327+0000 7f73469fe700 1 --2- 172.21.15.184:0/3696027193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73400f6510 0x7f734006b9a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f7334005010 tx=0x7f733400b040).stop 2024-02-19T22:12:45.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.327+0000 7f73469fe700 1 -- 172.21.15.184:0/3696027193 shutdown_connections 2024-02-19T22:12:45.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.327+0000 7f73469fe700 1 --2- 172.21.15.184:0/3696027193 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f732802ff10 0x7f73280323d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:45.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.327+0000 7f73469fe700 1 --2- 172.21.15.184:0/3696027193 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73400f6510 0x7f734006b9a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:45.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.327+0000 7f73469fe700 1 -- 172.21.15.184:0/3696027193 >> 172.21.15.184:0/3696027193 conn(0x7f73400f1c70 msgr2=0x7f73400f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:45.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.327+0000 7f73469fe700 1 -- 172.21.15.184:0/3696027193 shutdown_connections 2024-02-19T22:12:45.329 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:45.327+0000 7f73469fe700 1 -- 172.21.15.184:0/3696027193 wait complete. 2024-02-19T22:12:45.329 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:46.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:43.866741+0000 mgr.smithi176.faoogw (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:46.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:45.326701+0000 mon.smithi176 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.184:0/3696027193' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:46.943 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:46.944 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:48.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:45.867040+0000 mgr.smithi176.faoogw (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:48.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.719+0000 7fefe1a4c700 1 -- 172.21.15.184:0/4109744872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefdc0f5410 msgr2=0x7fefdc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:48.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.719+0000 7fefe1a4c700 1 --2- 172.21.15.184:0/4109744872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefdc0f5410 0x7fefdc0f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fefd0009fb0 tx=0x7fefd0009b30).stop 2024-02-19T22:12:48.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.719+0000 7fefe1a4c700 1 -- 172.21.15.184:0/4109744872 shutdown_connections 2024-02-19T22:12:48.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.719+0000 7fefe1a4c700 1 --2- 172.21.15.184:0/4109744872 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefdc0f5410 0x7fefdc0f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:48.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.719+0000 7fefe1a4c700 1 -- 172.21.15.184:0/4109744872 >> 172.21.15.184:0/4109744872 conn(0x7fefdc0f0b70 msgr2=0x7fefdc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:48.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.719+0000 7fefe1a4c700 1 -- 172.21.15.184:0/4109744872 shutdown_connections 2024-02-19T22:12:48.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.719+0000 7fefe1a4c700 1 -- 172.21.15.184:0/4109744872 wait complete. 2024-02-19T22:12:48.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.720+0000 7fefe1a4c700 1 Processor -- start 2024-02-19T22:12:48.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.720+0000 7fefe1a4c700 1 -- start start 2024-02-19T22:12:48.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.720+0000 7fefe1a4c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefdc0f5410 0x7fefdc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:48.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.720+0000 7fefe1a4c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc106cd0 con 0x7fefdc0f5410 2024-02-19T22:12:48.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.720+0000 7fefdaffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefdc0f5410 0x7fefdc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:48.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.720+0000 7fefdaffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefdc0f5410 0x7fefdc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:59512/0 (socket says 172.21.15.184:59512) 2024-02-19T22:12:48.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.720+0000 7fefdaffd700 1 -- 172.21.15.184:0/3941248929 learned_addr learned my addr 172.21.15.184:0/3941248929 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:48.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.720+0000 7fefdaffd700 1 -- 172.21.15.184:0/3941248929 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd0005040 con 0x7fefdc0f5410 2024-02-19T22:12:48.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.720+0000 7fefdaffd700 1 --2- 172.21.15.184:0/3941248929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefdc0f5410 0x7fefdc1067e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fefd0005010 tx=0x7fefd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:48.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.721+0000 7fefc37fe700 1 -- 172.21.15.184:0/3941248929 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefd0009320 con 0x7fefdc0f5410 2024-02-19T22:12:48.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.721+0000 7fefe1a4c700 1 -- 172.21.15.184:0/3941248929 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefdc106ed0 con 0x7fefdc0f5410 2024-02-19T22:12:48.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.721+0000 7fefc37fe700 1 -- 172.21.15.184:0/3941248929 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefd0013380 con 0x7fefdc0f5410 2024-02-19T22:12:48.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.721+0000 7fefc37fe700 1 -- 172.21.15.184:0/3941248929 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefd0014e50 con 0x7fefdc0f5410 2024-02-19T22:12:48.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.721+0000 7fefe1a4c700 1 -- 172.21.15.184:0/3941248929 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefdc107360 con 0x7fefdc0f5410 2024-02-19T22:12:48.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.721+0000 7fefc37fe700 1 -- 172.21.15.184:0/3941248929 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefd00263a0 con 0x7fefdc0f5410 2024-02-19T22:12:48.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.721+0000 7fefe1a4c700 1 -- 172.21.15.184:0/3941248929 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefdc045ad0 con 0x7fefdc0f5410 2024-02-19T22:12:48.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.721+0000 7fefc37fe700 1 --2- 172.21.15.184:0/3941248929 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fefc402ff10 0x7fefc40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:48.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.722+0000 7fefc37fe700 1 -- 172.21.15.184:0/3941248929 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefd0040ff0 con 0x7fefdc0f5410 2024-02-19T22:12:48.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.722+0000 7fefda7fc700 1 --2- 172.21.15.184:0/3941248929 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fefc402ff10 0x7fefc40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:48.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.722+0000 7fefda7fc700 1 --2- 172.21.15.184:0/3941248929 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fefc402ff10 0x7fefc40323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fefcc0096a0 tx=0x7fefcc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:48.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.727+0000 7fefc37fe700 1 -- 172.21.15.184:0/3941248929 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefd00126f0 con 0x7fefdc0f5410 2024-02-19T22:12:48.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.934+0000 7fefe1a4c700 1 -- 172.21.15.184:0/3941248929 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefdc0fafa0 con 0x7fefdc0f5410 2024-02-19T22:12:48.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.936+0000 7fefc37fe700 1 -- 172.21.15.184:0/3941248929 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fefd0012330 con 0x7fefdc0f5410 2024-02-19T22:12:48.937 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:48.937 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:48.939 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.936+0000 7fefe1a4c700 1 -- 172.21.15.184:0/3941248929 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fefc402ff10 msgr2=0x7fefc40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:48.939 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.936+0000 7fefe1a4c700 1 --2- 172.21.15.184:0/3941248929 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fefc402ff10 0x7fefc40323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fefcc0096a0 tx=0x7fefcc006b40).stop 2024-02-19T22:12:48.939 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.936+0000 7fefe1a4c700 1 -- 172.21.15.184:0/3941248929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefdc0f5410 msgr2=0x7fefdc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:48.939 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.936+0000 7fefe1a4c700 1 --2- 172.21.15.184:0/3941248929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefdc0f5410 0x7fefdc1067e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fefd0005010 tx=0x7fefd000b040).stop 2024-02-19T22:12:48.939 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.936+0000 7fefe1a4c700 1 -- 172.21.15.184:0/3941248929 shutdown_connections 2024-02-19T22:12:48.940 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.936+0000 7fefe1a4c700 1 --2- 172.21.15.184:0/3941248929 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fefc402ff10 0x7fefc40323d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:48.940 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.936+0000 7fefe1a4c700 1 --2- 172.21.15.184:0/3941248929 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fefdc0f5410 0x7fefdc1067e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:48.940 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.936+0000 7fefe1a4c700 1 -- 172.21.15.184:0/3941248929 >> 172.21.15.184:0/3941248929 conn(0x7fefdc0f0b70 msgr2=0x7fefdc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:48.940 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.936+0000 7fefe1a4c700 1 -- 172.21.15.184:0/3941248929 shutdown_connections 2024-02-19T22:12:48.940 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:48.937+0000 7fefe1a4c700 1 -- 172.21.15.184:0/3941248929 wait complete. 2024-02-19T22:12:48.941 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:50.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:49 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:47.867287+0000 mgr.smithi176.faoogw (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:50.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:49 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:48.936202+0000 mon.smithi176 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.184:0/3941248929' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:50.619 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:50.620 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:52.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:51 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:49.867652+0000 mgr.smithi176.faoogw (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:52.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.480+0000 7f645b117700 1 -- 172.21.15.184:0/1144575587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64540f5410 msgr2=0x7f64540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:52.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.480+0000 7f645b117700 1 --2- 172.21.15.184:0/1144575587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64540f5410 0x7f64540f5830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f6448009fb0 tx=0x7f6448009b30).stop 2024-02-19T22:12:52.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.480+0000 7f645b117700 1 -- 172.21.15.184:0/1144575587 shutdown_connections 2024-02-19T22:12:52.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.480+0000 7f645b117700 1 --2- 172.21.15.184:0/1144575587 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64540f5410 0x7f64540f5830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:52.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.480+0000 7f645b117700 1 -- 172.21.15.184:0/1144575587 >> 172.21.15.184:0/1144575587 conn(0x7f64540f0b70 msgr2=0x7f64540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:52.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.480+0000 7f645b117700 1 -- 172.21.15.184:0/1144575587 shutdown_connections 2024-02-19T22:12:52.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.480+0000 7f645b117700 1 -- 172.21.15.184:0/1144575587 wait complete. 2024-02-19T22:12:52.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.481+0000 7f645b117700 1 Processor -- start 2024-02-19T22:12:52.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.481+0000 7f645b117700 1 -- start start 2024-02-19T22:12:52.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.481+0000 7f645b117700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64540f5410 0x7f6454105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:52.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.481+0000 7f645b117700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6454105c10 con 0x7f64540f5410 2024-02-19T22:12:52.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.482+0000 7f6458eb3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64540f5410 0x7f6454105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:52.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.482+0000 7f6458eb3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64540f5410 0x7f6454105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:59530/0 (socket says 172.21.15.184:59530) 2024-02-19T22:12:52.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.482+0000 7f6458eb3700 1 -- 172.21.15.184:0/3989218135 learned_addr learned my addr 172.21.15.184:0/3989218135 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:52.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.482+0000 7f6458eb3700 1 -- 172.21.15.184:0/3989218135 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6448005040 con 0x7f64540f5410 2024-02-19T22:12:52.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.482+0000 7f6458eb3700 1 --2- 172.21.15.184:0/3989218135 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64540f5410 0x7f6454105720 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f6448005010 tx=0x7f644800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:52.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.482+0000 7f64517fa700 1 -- 172.21.15.184:0/3989218135 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6448009320 con 0x7f64540f5410 2024-02-19T22:12:52.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.482+0000 7f645b117700 1 -- 172.21.15.184:0/3989218135 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6454105e10 con 0x7f64540f5410 2024-02-19T22:12:52.483 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.482+0000 7f64517fa700 1 -- 172.21.15.184:0/3989218135 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6448013380 con 0x7f64540f5410 2024-02-19T22:12:52.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.482+0000 7f64517fa700 1 -- 172.21.15.184:0/3989218135 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6448014e50 con 0x7f64540f5410 2024-02-19T22:12:52.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.482+0000 7f645b117700 1 -- 172.21.15.184:0/3989218135 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64541062a0 con 0x7f64540f5410 2024-02-19T22:12:52.484 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.483+0000 7f64517fa700 1 -- 172.21.15.184:0/3989218135 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64480095f0 con 0x7f64540f5410 2024-02-19T22:12:52.485 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.483+0000 7f645b117700 1 -- 172.21.15.184:0/3989218135 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6454045ad0 con 0x7f64540f5410 2024-02-19T22:12:52.485 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.483+0000 7f64517fa700 1 --2- 172.21.15.184:0/3989218135 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f644402ff10 0x7f64440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:52.485 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.483+0000 7f64517fa700 1 -- 172.21.15.184:0/3989218135 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6448040f80 con 0x7f64540f5410 2024-02-19T22:12:52.485 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.483+0000 7f6453fff700 1 --2- 172.21.15.184:0/3989218135 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f644402ff10 0x7f64440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:52.485 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.484+0000 7f6453fff700 1 --2- 172.21.15.184:0/3989218135 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f644402ff10 0x7f64440323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f64400096a0 tx=0x7f6440006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:52.488 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.488+0000 7f64517fa700 1 -- 172.21.15.184:0/3989218135 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6448014420 con 0x7f64540f5410 2024-02-19T22:12:52.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.693+0000 7f645b117700 1 -- 172.21.15.184:0/3989218135 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64541070f0 con 0x7f64540f5410 2024-02-19T22:12:52.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.694+0000 7f64517fa700 1 -- 172.21.15.184:0/3989218135 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6448010230 con 0x7f64540f5410 2024-02-19T22:12:52.694 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:52.695 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:52.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.696+0000 7f645b117700 1 -- 172.21.15.184:0/3989218135 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f644402ff10 msgr2=0x7f64440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:52.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.696+0000 7f645b117700 1 --2- 172.21.15.184:0/3989218135 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f644402ff10 0x7f64440323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f64400096a0 tx=0x7f6440006b40).stop 2024-02-19T22:12:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.696+0000 7f645b117700 1 -- 172.21.15.184:0/3989218135 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64540f5410 msgr2=0x7f6454105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.696+0000 7f645b117700 1 --2- 172.21.15.184:0/3989218135 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64540f5410 0x7f6454105720 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f6448005010 tx=0x7f644800b040).stop 2024-02-19T22:12:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.696+0000 7f645b117700 1 -- 172.21.15.184:0/3989218135 shutdown_connections 2024-02-19T22:12:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.696+0000 7f645b117700 1 --2- 172.21.15.184:0/3989218135 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f644402ff10 0x7f64440323d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.696+0000 7f645b117700 1 --2- 172.21.15.184:0/3989218135 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64540f5410 0x7f6454105720 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.696+0000 7f645b117700 1 -- 172.21.15.184:0/3989218135 >> 172.21.15.184:0/3989218135 conn(0x7f64540f0b70 msgr2=0x7f64540f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.696+0000 7f645b117700 1 -- 172.21.15.184:0/3989218135 shutdown_connections 2024-02-19T22:12:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:52.696+0000 7f645b117700 1 -- 172.21.15.184:0/3989218135 wait complete. 2024-02-19T22:12:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:53.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:52 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:52.694213+0000 mon.smithi176 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.184:0/3989218135' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:54.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:51.868029+0000 mgr.smithi176.faoogw (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:54.307 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:54.307 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:56.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:55 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:53.868357+0000 mgr.smithi176.faoogw (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:56.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.247+0000 7f304406d700 1 -- 172.21.15.184:0/2039524732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c0f5410 msgr2=0x7f303c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:56.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.247+0000 7f304406d700 1 --2- 172.21.15.184:0/2039524732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c0f5410 0x7f303c0f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f302c009fb0 tx=0x7f302c009b30).stop 2024-02-19T22:12:56.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.247+0000 7f304406d700 1 -- 172.21.15.184:0/2039524732 shutdown_connections 2024-02-19T22:12:56.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.247+0000 7f304406d700 1 --2- 172.21.15.184:0/2039524732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c0f5410 0x7f303c0f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:56.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.247+0000 7f304406d700 1 -- 172.21.15.184:0/2039524732 >> 172.21.15.184:0/2039524732 conn(0x7f303c0f0b70 msgr2=0x7f303c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:56.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.247+0000 7f304406d700 1 -- 172.21.15.184:0/2039524732 shutdown_connections 2024-02-19T22:12:56.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.247+0000 7f304406d700 1 -- 172.21.15.184:0/2039524732 wait complete. 2024-02-19T22:12:56.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.248+0000 7f304406d700 1 Processor -- start 2024-02-19T22:12:56.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.248+0000 7f304406d700 1 -- start start 2024-02-19T22:12:56.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.248+0000 7f304406d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c0f5410 0x7f303c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:56.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.248+0000 7f304406d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c106cc0 con 0x7f303c0f5410 2024-02-19T22:12:56.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.248+0000 7f3041e09700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c0f5410 0x7f303c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:56.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.248+0000 7f3041e09700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c0f5410 0x7f303c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:59542/0 (socket says 172.21.15.184:59542) 2024-02-19T22:12:56.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.248+0000 7f3041e09700 1 -- 172.21.15.184:0/2224579030 learned_addr learned my addr 172.21.15.184:0/2224579030 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:56.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.248+0000 7f3041e09700 1 -- 172.21.15.184:0/2224579030 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c005040 con 0x7f303c0f5410 2024-02-19T22:12:56.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.249+0000 7f3041e09700 1 --2- 172.21.15.184:0/2224579030 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c0f5410 0x7f303c1067d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f302c0089f0 tx=0x7f302c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:56.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.249+0000 7f30327fc700 1 -- 172.21.15.184:0/2224579030 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f302c009320 con 0x7f303c0f5410 2024-02-19T22:12:56.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.249+0000 7f304406d700 1 -- 172.21.15.184:0/2224579030 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303c106ec0 con 0x7f303c0f5410 2024-02-19T22:12:56.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.249+0000 7f30327fc700 1 -- 172.21.15.184:0/2224579030 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f302c013380 con 0x7f303c0f5410 2024-02-19T22:12:56.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.249+0000 7f30327fc700 1 -- 172.21.15.184:0/2224579030 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f302c01ed80 con 0x7f303c0f5410 2024-02-19T22:12:56.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.249+0000 7f304406d700 1 -- 172.21.15.184:0/2224579030 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303c107350 con 0x7f303c0f5410 2024-02-19T22:12:56.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.249+0000 7f30327fc700 1 -- 172.21.15.184:0/2224579030 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f302c014420 con 0x7f303c0f5410 2024-02-19T22:12:56.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.250+0000 7f30327fc700 1 --2- 172.21.15.184:0/2224579030 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f302802ff60 0x7f3028032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:56.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.250+0000 7f304406d700 1 -- 172.21.15.184:0/2224579030 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303c045ad0 con 0x7f303c0f5410 2024-02-19T22:12:56.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.250+0000 7f30327fc700 1 -- 172.21.15.184:0/2224579030 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f302c0411b0 con 0x7f303c0f5410 2024-02-19T22:12:56.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.250+0000 7f3041608700 1 --2- 172.21.15.184:0/2224579030 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f302802ff60 0x7f3028032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:56.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.250+0000 7f3041608700 1 --2- 172.21.15.184:0/2224579030 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f302802ff60 0x7f3028032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f30380096a0 tx=0x7f3038006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:56.259 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.254+0000 7f30327fc700 1 -- 172.21.15.184:0/2224579030 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f302c00faf0 con 0x7f303c0f5410 2024-02-19T22:12:56.460 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.458+0000 7f304406d700 1 -- 172.21.15.184:0/2224579030 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f303c108200 con 0x7f303c0f5410 2024-02-19T22:12:56.460 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.459+0000 7f30327fc700 1 -- 172.21.15.184:0/2224579030 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f302c02c2d0 con 0x7f303c0f5410 2024-02-19T22:12:56.461 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:12:56.461 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:12:56.462 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.461+0000 7f304406d700 1 -- 172.21.15.184:0/2224579030 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f302802ff60 msgr2=0x7f3028032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:56.462 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.461+0000 7f304406d700 1 --2- 172.21.15.184:0/2224579030 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f302802ff60 0x7f3028032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f30380096a0 tx=0x7f3038006b40).stop 2024-02-19T22:12:56.462 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.461+0000 7f304406d700 1 -- 172.21.15.184:0/2224579030 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c0f5410 msgr2=0x7f303c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:56.462 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.461+0000 7f304406d700 1 --2- 172.21.15.184:0/2224579030 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c0f5410 0x7f303c1067d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f302c0089f0 tx=0x7f302c00b040).stop 2024-02-19T22:12:56.462 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.461+0000 7f304406d700 1 -- 172.21.15.184:0/2224579030 shutdown_connections 2024-02-19T22:12:56.462 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.461+0000 7f304406d700 1 --2- 172.21.15.184:0/2224579030 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f302802ff60 0x7f3028032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:56.462 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.461+0000 7f304406d700 1 --2- 172.21.15.184:0/2224579030 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f303c0f5410 0x7f303c1067d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:56.462 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.461+0000 7f304406d700 1 -- 172.21.15.184:0/2224579030 >> 172.21.15.184:0/2224579030 conn(0x7f303c0f0b70 msgr2=0x7f303c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:56.462 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.461+0000 7f304406d700 1 -- 172.21.15.184:0/2224579030 shutdown_connections 2024-02-19T22:12:56.463 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:56.461+0000 7f304406d700 1 -- 172.21.15.184:0/2224579030 wait complete. 2024-02-19T22:12:56.463 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:12:57.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:56 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:12:56.460452+0000 mon.smithi176 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.184:0/2224579030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:12:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:55.868716+0000 mgr.smithi176.faoogw (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:12:58.138 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:12:58.139 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:12:59.993 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.991+0000 7f5ab3950700 1 -- 172.21.15.184:0/3173932412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5aac0f5410 msgr2=0x7f5aac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:59.994 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.991+0000 7f5ab3950700 1 --2- 172.21.15.184:0/3173932412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5aac0f5410 0x7f5aac0f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f5a9c009fb0 tx=0x7f5a9c009b30).stop 2024-02-19T22:12:59.994 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.991+0000 7f5ab3950700 1 -- 172.21.15.184:0/3173932412 shutdown_connections 2024-02-19T22:12:59.994 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.991+0000 7f5ab3950700 1 --2- 172.21.15.184:0/3173932412 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5aac0f5410 0x7f5aac0f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:59.994 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.991+0000 7f5ab3950700 1 -- 172.21.15.184:0/3173932412 >> 172.21.15.184:0/3173932412 conn(0x7f5aac0f0b70 msgr2=0x7f5aac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:59.994 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.991+0000 7f5ab3950700 1 -- 172.21.15.184:0/3173932412 shutdown_connections 2024-02-19T22:12:59.994 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.991+0000 7f5ab3950700 1 -- 172.21.15.184:0/3173932412 wait complete. 2024-02-19T22:12:59.994 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.992+0000 7f5ab3950700 1 Processor -- start 2024-02-19T22:12:59.994 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.992+0000 7f5ab3950700 1 -- start start 2024-02-19T22:12:59.994 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.992+0000 7f5ab3950700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5aac0f5410 0x7f5aac1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:59.994 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.992+0000 7f5ab3950700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac105bd0 con 0x7f5aac0f5410 2024-02-19T22:12:59.994 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.992+0000 7f5ab16ec700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5aac0f5410 0x7f5aac1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:59.994 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.992+0000 7f5ab16ec700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5aac0f5410 0x7f5aac1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:60134/0 (socket says 172.21.15.184:60134) 2024-02-19T22:12:59.995 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.992+0000 7f5ab16ec700 1 -- 172.21.15.184:0/4043012343 learned_addr learned my addr 172.21.15.184:0/4043012343 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:12:59.995 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.992+0000 7f5ab16ec700 1 -- 172.21.15.184:0/4043012343 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9c005040 con 0x7f5aac0f5410 2024-02-19T22:12:59.995 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.992+0000 7f5ab16ec700 1 --2- 172.21.15.184:0/4043012343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5aac0f5410 0x7f5aac1056e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f5a9c005010 tx=0x7f5a9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:59.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.993+0000 7f5aa1ffb700 1 -- 172.21.15.184:0/4043012343 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a9c009320 con 0x7f5aac0f5410 2024-02-19T22:12:59.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.993+0000 7f5ab3950700 1 -- 172.21.15.184:0/4043012343 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aac105dd0 con 0x7f5aac0f5410 2024-02-19T22:12:59.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.993+0000 7f5aa1ffb700 1 -- 172.21.15.184:0/4043012343 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a9c013380 con 0x7f5aac0f5410 2024-02-19T22:12:59.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.993+0000 7f5aa1ffb700 1 -- 172.21.15.184:0/4043012343 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a9c014e50 con 0x7f5aac0f5410 2024-02-19T22:12:59.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.993+0000 7f5ab3950700 1 -- 172.21.15.184:0/4043012343 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aac106260 con 0x7f5aac0f5410 2024-02-19T22:12:59.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.994+0000 7f5aa1ffb700 1 -- 172.21.15.184:0/4043012343 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a9c0095f0 con 0x7f5aac0f5410 2024-02-19T22:12:59.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.994+0000 7f5ab3950700 1 -- 172.21.15.184:0/4043012343 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aac045ad0 con 0x7f5aac0f5410 2024-02-19T22:12:59.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.994+0000 7f5aa1ffb700 1 --2- 172.21.15.184:0/4043012343 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f5a9802ff10 0x7f5a980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:59.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.994+0000 7f5aa1ffb700 1 -- 172.21.15.184:0/4043012343 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a9c040e90 con 0x7f5aac0f5410 2024-02-19T22:12:59.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.994+0000 7f5ab0eeb700 1 --2- 172.21.15.184:0/4043012343 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f5a9802ff10 0x7f5a980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:59.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.994+0000 7f5ab0eeb700 1 --2- 172.21.15.184:0/4043012343 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f5a9802ff10 0x7f5a980323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f5aa80096a0 tx=0x7f5aa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:59.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:12:59.998+0000 7f5aa1ffb700 1 -- 172.21.15.184:0/4043012343 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a9c014420 con 0x7f5aac0f5410 2024-02-19T22:13:00.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:12:59 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:57.869004+0000 mgr.smithi176.faoogw (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:00.204 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:00.203+0000 7f5ab3950700 1 -- 172.21.15.184:0/4043012343 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5aac107110 con 0x7f5aac0f5410 2024-02-19T22:13:00.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:00.204+0000 7f5aa1ffb700 1 -- 172.21.15.184:0/4043012343 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5a9c010230 con 0x7f5aac0f5410 2024-02-19T22:13:00.205 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:00.206 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:00.207 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:00.206+0000 7f5ab3950700 1 -- 172.21.15.184:0/4043012343 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f5a9802ff10 msgr2=0x7f5a980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:00.207 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:00.206+0000 7f5ab3950700 1 --2- 172.21.15.184:0/4043012343 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f5a9802ff10 0x7f5a980323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f5aa80096a0 tx=0x7f5aa8006b40).stop 2024-02-19T22:13:00.207 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:00.206+0000 7f5ab3950700 1 -- 172.21.15.184:0/4043012343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5aac0f5410 msgr2=0x7f5aac1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:00.207 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:00.206+0000 7f5ab3950700 1 --2- 172.21.15.184:0/4043012343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5aac0f5410 0x7f5aac1056e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f5a9c005010 tx=0x7f5a9c00b040).stop 2024-02-19T22:13:00.207 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:00.206+0000 7f5ab3950700 1 -- 172.21.15.184:0/4043012343 shutdown_connections 2024-02-19T22:13:00.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:00.206+0000 7f5ab3950700 1 --2- 172.21.15.184:0/4043012343 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f5a9802ff10 0x7f5a980323d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:00.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:00.206+0000 7f5ab3950700 1 --2- 172.21.15.184:0/4043012343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5aac0f5410 0x7f5aac1056e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:00.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:00.206+0000 7f5ab3950700 1 -- 172.21.15.184:0/4043012343 >> 172.21.15.184:0/4043012343 conn(0x7f5aac0f0b70 msgr2=0x7f5aac0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:00.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:00.206+0000 7f5ab3950700 1 -- 172.21.15.184:0/4043012343 shutdown_connections 2024-02-19T22:13:00.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:00.206+0000 7f5ab3950700 1 -- 172.21.15.184:0/4043012343 wait complete. 2024-02-19T22:13:00.208 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:01.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:00 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:00.205290+0000 mon.smithi176 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.184:0/4043012343' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:01.896 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:01.896 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:02.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:12:59.869254+0000 mgr.smithi176.faoogw (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:03.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.715+0000 7f6463fff700 1 -- 172.21.15.184:0/35498368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64640f51a0 msgr2=0x7f64640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:03.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.715+0000 7f6463fff700 1 --2- 172.21.15.184:0/35498368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64640f51a0 0x7f64640f55c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f6454009fb0 tx=0x7f6454009b30).stop 2024-02-19T22:13:03.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.715+0000 7f6463fff700 1 -- 172.21.15.184:0/35498368 shutdown_connections 2024-02-19T22:13:03.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.715+0000 7f6463fff700 1 --2- 172.21.15.184:0/35498368 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64640f51a0 0x7f64640f55c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:03.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.715+0000 7f6463fff700 1 -- 172.21.15.184:0/35498368 >> 172.21.15.184:0/35498368 conn(0x7f64640f09a0 msgr2=0x7f64640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:03.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.715+0000 7f6463fff700 1 -- 172.21.15.184:0/35498368 shutdown_connections 2024-02-19T22:13:03.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.715+0000 7f6463fff700 1 -- 172.21.15.184:0/35498368 wait complete. 2024-02-19T22:13:03.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.716+0000 7f6463fff700 1 Processor -- start 2024-02-19T22:13:03.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.716+0000 7f6463fff700 1 -- start start 2024-02-19T22:13:03.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.716+0000 7f6463fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64640f51a0 0x7f64640fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:03.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.716+0000 7f6463fff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64640fb6f0 con 0x7f64640f51a0 2024-02-19T22:13:03.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.716+0000 7f6462ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64640f51a0 0x7f64640fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:03.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.717+0000 7f6462ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64640f51a0 0x7f64640fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:60152/0 (socket says 172.21.15.184:60152) 2024-02-19T22:13:03.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.717+0000 7f6462ffd700 1 -- 172.21.15.184:0/2649961831 learned_addr learned my addr 172.21.15.184:0/2649961831 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:03.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.717+0000 7f6462ffd700 1 -- 172.21.15.184:0/2649961831 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6454005040 con 0x7f64640f51a0 2024-02-19T22:13:03.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.717+0000 7f6462ffd700 1 --2- 172.21.15.184:0/2649961831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64640f51a0 0x7f64640fd270 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f64540089f0 tx=0x7f645400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:03.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.717+0000 7f644b7fe700 1 -- 172.21.15.184:0/2649961831 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6454009320 con 0x7f64640f51a0 2024-02-19T22:13:03.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.717+0000 7f6463fff700 1 -- 172.21.15.184:0/2649961831 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64640fb890 con 0x7f64640f51a0 2024-02-19T22:13:03.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.717+0000 7f644b7fe700 1 -- 172.21.15.184:0/2649961831 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6454013380 con 0x7f64640f51a0 2024-02-19T22:13:03.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.717+0000 7f644b7fe700 1 -- 172.21.15.184:0/2649961831 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6454014e20 con 0x7f64640f51a0 2024-02-19T22:13:03.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.717+0000 7f6463fff700 1 -- 172.21.15.184:0/2649961831 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64640fbd20 con 0x7f64640f51a0 2024-02-19T22:13:03.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.718+0000 7f644b7fe700 1 -- 172.21.15.184:0/2649961831 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64540095f0 con 0x7f64640f51a0 2024-02-19T22:13:03.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.718+0000 7f6463fff700 1 -- 172.21.15.184:0/2649961831 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6464059100 con 0x7f64640f51a0 2024-02-19T22:13:03.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.718+0000 7f644b7fe700 1 --2- 172.21.15.184:0/2649961831 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f644c02ff10 0x7f644c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:03.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.718+0000 7f644b7fe700 1 -- 172.21.15.184:0/2649961831 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6454040e90 con 0x7f64640f51a0 2024-02-19T22:13:03.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.718+0000 7f64627fc700 1 --2- 172.21.15.184:0/2649961831 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f644c02ff10 0x7f644c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:03.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.719+0000 7f64627fc700 1 --2- 172.21.15.184:0/2649961831 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f644c02ff10 0x7f644c0323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f64580096a0 tx=0x7f6458006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:03.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.722+0000 7f644b7fe700 1 -- 172.21.15.184:0/2649961831 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6454014420 con 0x7f64640f51a0 2024-02-19T22:13:03.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.927+0000 7f6463fff700 1 -- 172.21.15.184:0/2649961831 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64640fad30 con 0x7f64640f51a0 2024-02-19T22:13:03.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.928+0000 7f644b7fe700 1 -- 172.21.15.184:0/2649961831 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f645401d980 con 0x7f64640f51a0 2024-02-19T22:13:03.930 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:03.930 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:03.931 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.930+0000 7f6463fff700 1 -- 172.21.15.184:0/2649961831 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f644c02ff10 msgr2=0x7f644c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:03.931 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.930+0000 7f6463fff700 1 --2- 172.21.15.184:0/2649961831 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f644c02ff10 0x7f644c0323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f64580096a0 tx=0x7f6458006b40).stop 2024-02-19T22:13:03.931 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.930+0000 7f6463fff700 1 -- 172.21.15.184:0/2649961831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64640f51a0 msgr2=0x7f64640fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:03.931 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.930+0000 7f6463fff700 1 --2- 172.21.15.184:0/2649961831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64640f51a0 0x7f64640fd270 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f64540089f0 tx=0x7f645400b040).stop 2024-02-19T22:13:03.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.930+0000 7f6463fff700 1 -- 172.21.15.184:0/2649961831 shutdown_connections 2024-02-19T22:13:03.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.930+0000 7f6463fff700 1 --2- 172.21.15.184:0/2649961831 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f644c02ff10 0x7f644c0323d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:03.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.930+0000 7f6463fff700 1 --2- 172.21.15.184:0/2649961831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64640f51a0 0x7f64640fd270 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:03.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.930+0000 7f6463fff700 1 -- 172.21.15.184:0/2649961831 >> 172.21.15.184:0/2649961831 conn(0x7f64640f09a0 msgr2=0x7f646405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:03.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.930+0000 7f6463fff700 1 -- 172.21.15.184:0/2649961831 shutdown_connections 2024-02-19T22:13:03.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:03.930+0000 7f6463fff700 1 -- 172.21.15.184:0/2649961831 wait complete. 2024-02-19T22:13:03.932 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:04.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:01.869564+0000 mgr.smithi176.faoogw (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:05.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:04 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:03.929674+0000 mon.smithi176 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.184:0/2649961831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:05.552 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:05.553 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:06.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:03.869762+0000 mgr.smithi176.faoogw (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:07.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.349+0000 7f6813591700 1 -- 172.21.15.184:0/3687881148 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f680c0f5410 msgr2=0x7f680c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:07.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.349+0000 7f6813591700 1 --2- 172.21.15.184:0/3687881148 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f680c0f5410 0x7f680c0f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f6808009fb0 tx=0x7f6808009b30).stop 2024-02-19T22:13:07.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.349+0000 7f6813591700 1 -- 172.21.15.184:0/3687881148 shutdown_connections 2024-02-19T22:13:07.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.349+0000 7f6813591700 1 --2- 172.21.15.184:0/3687881148 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f680c0f5410 0x7f680c0f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:07.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.349+0000 7f6813591700 1 -- 172.21.15.184:0/3687881148 >> 172.21.15.184:0/3687881148 conn(0x7f680c0f0b70 msgr2=0x7f680c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:07.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.349+0000 7f6813591700 1 -- 172.21.15.184:0/3687881148 shutdown_connections 2024-02-19T22:13:07.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.349+0000 7f6813591700 1 -- 172.21.15.184:0/3687881148 wait complete. 2024-02-19T22:13:07.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.349+0000 7f6813591700 1 Processor -- start 2024-02-19T22:13:07.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.349+0000 7f6813591700 1 -- start start 2024-02-19T22:13:07.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.349+0000 7f6813591700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f680c0f5410 0x7f680c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:07.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.350+0000 7f6813591700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c106cd0 con 0x7f680c0f5410 2024-02-19T22:13:07.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.350+0000 7f681132d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f680c0f5410 0x7f680c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:07.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.350+0000 7f681132d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f680c0f5410 0x7f680c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:48442/0 (socket says 172.21.15.184:48442) 2024-02-19T22:13:07.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.350+0000 7f681132d700 1 -- 172.21.15.184:0/1288712199 learned_addr learned my addr 172.21.15.184:0/1288712199 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:07.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.350+0000 7f681132d700 1 -- 172.21.15.184:0/1288712199 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6808005040 con 0x7f680c0f5410 2024-02-19T22:13:07.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.350+0000 7f681132d700 1 --2- 172.21.15.184:0/1288712199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f680c0f5410 0x7f680c1067e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f6808005010 tx=0x7f680800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:07.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.350+0000 7f6801ffb700 1 -- 172.21.15.184:0/1288712199 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6808009320 con 0x7f680c0f5410 2024-02-19T22:13:07.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.350+0000 7f6813591700 1 -- 172.21.15.184:0/1288712199 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680c106ed0 con 0x7f680c0f5410 2024-02-19T22:13:07.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.350+0000 7f6801ffb700 1 -- 172.21.15.184:0/1288712199 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6808013380 con 0x7f680c0f5410 2024-02-19T22:13:07.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.350+0000 7f6801ffb700 1 -- 172.21.15.184:0/1288712199 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6808014e50 con 0x7f680c0f5410 2024-02-19T22:13:07.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.350+0000 7f6813591700 1 -- 172.21.15.184:0/1288712199 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680c107360 con 0x7f680c0f5410 2024-02-19T22:13:07.354 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.351+0000 7f6813591700 1 -- 172.21.15.184:0/1288712199 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680c045ad0 con 0x7f680c0f5410 2024-02-19T22:13:07.354 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.351+0000 7f6801ffb700 1 -- 172.21.15.184:0/1288712199 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68080095f0 con 0x7f680c0f5410 2024-02-19T22:13:07.354 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.351+0000 7f6801ffb700 1 --2- 172.21.15.184:0/1288712199 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f67f802ff10 0x7f67f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:07.354 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.352+0000 7f6801ffb700 1 -- 172.21.15.184:0/1288712199 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6808040fa0 con 0x7f680c0f5410 2024-02-19T22:13:07.354 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.352+0000 7f6810b2c700 1 --2- 172.21.15.184:0/1288712199 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f67f802ff10 0x7f67f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:07.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.352+0000 7f6810b2c700 1 --2- 172.21.15.184:0/1288712199 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f67f802ff10 0x7f67f80323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f67fc0096a0 tx=0x7f67fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:07.357 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.356+0000 7f6801ffb700 1 -- 172.21.15.184:0/1288712199 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6808014420 con 0x7f680c0f5410 2024-02-19T22:13:07.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.561+0000 7f6813591700 1 -- 172.21.15.184:0/1288712199 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f680c0fafa0 con 0x7f680c0f5410 2024-02-19T22:13:07.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.562+0000 7f6801ffb700 1 -- 172.21.15.184:0/1288712199 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6808010230 con 0x7f680c0f5410 2024-02-19T22:13:07.564 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:07.564 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:07.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.564+0000 7f6813591700 1 -- 172.21.15.184:0/1288712199 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f67f802ff10 msgr2=0x7f67f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:07.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.564+0000 7f6813591700 1 --2- 172.21.15.184:0/1288712199 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f67f802ff10 0x7f67f80323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f67fc0096a0 tx=0x7f67fc006b40).stop 2024-02-19T22:13:07.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.564+0000 7f6813591700 1 -- 172.21.15.184:0/1288712199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f680c0f5410 msgr2=0x7f680c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:07.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.564+0000 7f6813591700 1 --2- 172.21.15.184:0/1288712199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f680c0f5410 0x7f680c1067e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f6808005010 tx=0x7f680800b040).stop 2024-02-19T22:13:07.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.564+0000 7f6813591700 1 -- 172.21.15.184:0/1288712199 shutdown_connections 2024-02-19T22:13:07.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.564+0000 7f6813591700 1 --2- 172.21.15.184:0/1288712199 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f67f802ff10 0x7f67f80323d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:07.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.564+0000 7f6813591700 1 --2- 172.21.15.184:0/1288712199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f680c0f5410 0x7f680c1067e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:07.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.564+0000 7f6813591700 1 -- 172.21.15.184:0/1288712199 >> 172.21.15.184:0/1288712199 conn(0x7f680c0f0b70 msgr2=0x7f680c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:07.566 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.564+0000 7f6813591700 1 -- 172.21.15.184:0/1288712199 shutdown_connections 2024-02-19T22:13:07.567 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:07.564+0000 7f6813591700 1 -- 172.21.15.184:0/1288712199 wait complete. 2024-02-19T22:13:07.567 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:08.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:05.870003+0000 mgr.smithi176.faoogw (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:08.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:07.563672+0000 mon.smithi176 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.184:0/1288712199' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:09.195 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:09.195 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:10.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:07.870274+0000 mgr.smithi176.faoogw (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:11.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.139+0000 7f3415ec3700 1 -- 172.21.15.184:0/1804918641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3410060220 msgr2=0x7f34100f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:11.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.139+0000 7f3415ec3700 1 --2- 172.21.15.184:0/1804918641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3410060220 0x7f34100f9150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f3404009fb0 tx=0x7f3404009b30).stop 2024-02-19T22:13:11.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.139+0000 7f3415ec3700 1 -- 172.21.15.184:0/1804918641 shutdown_connections 2024-02-19T22:13:11.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.139+0000 7f3415ec3700 1 --2- 172.21.15.184:0/1804918641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3410060220 0x7f34100f9150 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:11.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.139+0000 7f3415ec3700 1 -- 172.21.15.184:0/1804918641 >> 172.21.15.184:0/1804918641 conn(0x7f34100f0b70 msgr2=0x7f34100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:11.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.139+0000 7f3415ec3700 1 -- 172.21.15.184:0/1804918641 shutdown_connections 2024-02-19T22:13:11.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.139+0000 7f3415ec3700 1 -- 172.21.15.184:0/1804918641 wait complete. 2024-02-19T22:13:11.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.140+0000 7f3415ec3700 1 Processor -- start 2024-02-19T22:13:11.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.140+0000 7f3415ec3700 1 -- start start 2024-02-19T22:13:11.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.140+0000 7f3415ec3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3410060220 0x7f34100f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:11.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.140+0000 7f3415ec3700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34100f5050 con 0x7f3410060220 2024-02-19T22:13:11.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.140+0000 7f340f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3410060220 0x7f34100f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:11.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.140+0000 7f340f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3410060220 0x7f34100f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:48464/0 (socket says 172.21.15.184:48464) 2024-02-19T22:13:11.144 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.140+0000 7f340f7fe700 1 -- 172.21.15.184:0/1482090145 learned_addr learned my addr 172.21.15.184:0/1482090145 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:11.144 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.140+0000 7f340f7fe700 1 -- 172.21.15.184:0/1482090145 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3404005040 con 0x7f3410060220 2024-02-19T22:13:11.144 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.140+0000 7f340f7fe700 1 --2- 172.21.15.184:0/1482090145 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3410060220 0x7f34100f6bd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f34040089f0 tx=0x7f340400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:11.144 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.141+0000 7f33f7fff700 1 -- 172.21.15.184:0/1482090145 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3404009320 con 0x7f3410060220 2024-02-19T22:13:11.144 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.141+0000 7f33f7fff700 1 -- 172.21.15.184:0/1482090145 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3404013380 con 0x7f3410060220 2024-02-19T22:13:11.144 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.141+0000 7f33f7fff700 1 -- 172.21.15.184:0/1482090145 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3404014e20 con 0x7f3410060220 2024-02-19T22:13:11.144 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.141+0000 7f3415ec3700 1 -- 172.21.15.184:0/1482090145 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34100f51f0 con 0x7f3410060220 2024-02-19T22:13:11.144 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.141+0000 7f3415ec3700 1 -- 172.21.15.184:0/1482090145 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34100f5680 con 0x7f3410060220 2024-02-19T22:13:11.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.141+0000 7f33f7fff700 1 -- 172.21.15.184:0/1482090145 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34040095f0 con 0x7f3410060220 2024-02-19T22:13:11.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.141+0000 7f3415ec3700 1 -- 172.21.15.184:0/1482090145 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3410059100 con 0x7f3410060220 2024-02-19T22:13:11.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.142+0000 7f33f7fff700 1 --2- 172.21.15.184:0/1482090145 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f33f802ff60 0x7f33f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:11.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.142+0000 7f33f7fff700 1 -- 172.21.15.184:0/1482090145 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3404040e40 con 0x7f3410060220 2024-02-19T22:13:11.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.142+0000 7f340effd700 1 --2- 172.21.15.184:0/1482090145 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f33f802ff60 0x7f33f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:11.145 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.142+0000 7f340effd700 1 --2- 172.21.15.184:0/1482090145 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f33f802ff60 0x7f33f8032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f34000096a0 tx=0x7f3400006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:11.148 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.147+0000 7f33f7fff700 1 -- 172.21.15.184:0/1482090145 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3404014420 con 0x7f3410060220 2024-02-19T22:13:11.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.350+0000 7f3415ec3700 1 -- 172.21.15.184:0/1482090145 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34100fc090 con 0x7f3410060220 2024-02-19T22:13:11.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.351+0000 7f33f7fff700 1 -- 172.21.15.184:0/1482090145 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3404010230 con 0x7f3410060220 2024-02-19T22:13:11.353 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:11.353 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:11.354 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.353+0000 7f3415ec3700 1 -- 172.21.15.184:0/1482090145 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f33f802ff60 msgr2=0x7f33f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:11.354 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.353+0000 7f3415ec3700 1 --2- 172.21.15.184:0/1482090145 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f33f802ff60 0x7f33f8032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f34000096a0 tx=0x7f3400006b40).stop 2024-02-19T22:13:11.354 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.353+0000 7f3415ec3700 1 -- 172.21.15.184:0/1482090145 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3410060220 msgr2=0x7f34100f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:11.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.353+0000 7f3415ec3700 1 --2- 172.21.15.184:0/1482090145 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3410060220 0x7f34100f6bd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f34040089f0 tx=0x7f340400b040).stop 2024-02-19T22:13:11.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.353+0000 7f3415ec3700 1 -- 172.21.15.184:0/1482090145 shutdown_connections 2024-02-19T22:13:11.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.353+0000 7f3415ec3700 1 --2- 172.21.15.184:0/1482090145 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f33f802ff60 0x7f33f8032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:11.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.353+0000 7f3415ec3700 1 --2- 172.21.15.184:0/1482090145 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3410060220 0x7f34100f6bd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:11.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.353+0000 7f3415ec3700 1 -- 172.21.15.184:0/1482090145 >> 172.21.15.184:0/1482090145 conn(0x7f34100f0b70 msgr2=0x7f34100f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:11.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.353+0000 7f3415ec3700 1 -- 172.21.15.184:0/1482090145 shutdown_connections 2024-02-19T22:13:11.355 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:11.353+0000 7f3415ec3700 1 -- 172.21.15.184:0/1482090145 wait complete. 2024-02-19T22:13:11.355 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:12.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:09.870551+0000 mgr.smithi176.faoogw (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:12.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:11.352571+0000 mon.smithi176 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.184:0/1482090145' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:12.999 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:12.999 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:14.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:13 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:11.870917+0000 mgr.smithi176.faoogw (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:14.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.849+0000 7ff24d9ec700 1 -- 172.21.15.184:0/1065761185 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2480f4860 msgr2=0x7ff2480f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:14.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.849+0000 7ff24d9ec700 1 --2- 172.21.15.184:0/1065761185 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2480f4860 0x7ff2480f4c80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7ff23c009fb0 tx=0x7ff23c009b30).stop 2024-02-19T22:13:14.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.849+0000 7ff24d9ec700 1 -- 172.21.15.184:0/1065761185 shutdown_connections 2024-02-19T22:13:14.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.849+0000 7ff24d9ec700 1 --2- 172.21.15.184:0/1065761185 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2480f4860 0x7ff2480f4c80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:14.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.849+0000 7ff24d9ec700 1 -- 172.21.15.184:0/1065761185 >> 172.21.15.184:0/1065761185 conn(0x7ff2480f02e0 msgr2=0x7ff2480f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:14.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.849+0000 7ff24d9ec700 1 -- 172.21.15.184:0/1065761185 shutdown_connections 2024-02-19T22:13:14.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.849+0000 7ff24d9ec700 1 -- 172.21.15.184:0/1065761185 wait complete. 2024-02-19T22:13:14.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.850+0000 7ff24d9ec700 1 Processor -- start 2024-02-19T22:13:14.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.850+0000 7ff24d9ec700 1 -- start start 2024-02-19T22:13:14.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.850+0000 7ff24d9ec700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2480f4860 0x7ff2481069c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:14.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.850+0000 7ff24d9ec700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff248106eb0 con 0x7ff2480f4860 2024-02-19T22:13:14.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.850+0000 7ff246ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2480f4860 0x7ff2481069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:14.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.850+0000 7ff246ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2480f4860 0x7ff2481069c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:48478/0 (socket says 172.21.15.184:48478) 2024-02-19T22:13:14.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.850+0000 7ff246ffd700 1 -- 172.21.15.184:0/2139776121 learned_addr learned my addr 172.21.15.184:0/2139776121 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:14.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.851+0000 7ff246ffd700 1 -- 172.21.15.184:0/2139776121 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23c005040 con 0x7ff2480f4860 2024-02-19T22:13:14.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.851+0000 7ff246ffd700 1 --2- 172.21.15.184:0/2139776121 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2480f4860 0x7ff2481069c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7ff23c0089f0 tx=0x7ff23c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:14.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.851+0000 7ff22f7fe700 1 -- 172.21.15.184:0/2139776121 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff23c009320 con 0x7ff2480f4860 2024-02-19T22:13:14.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.851+0000 7ff24d9ec700 1 -- 172.21.15.184:0/2139776121 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2481070b0 con 0x7ff2480f4860 2024-02-19T22:13:14.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.851+0000 7ff22f7fe700 1 -- 172.21.15.184:0/2139776121 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff23c013380 con 0x7ff2480f4860 2024-02-19T22:13:14.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.851+0000 7ff22f7fe700 1 -- 172.21.15.184:0/2139776121 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff23c014e20 con 0x7ff2480f4860 2024-02-19T22:13:14.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.851+0000 7ff24d9ec700 1 -- 172.21.15.184:0/2139776121 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff248107540 con 0x7ff2480f4860 2024-02-19T22:13:14.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.852+0000 7ff22f7fe700 1 -- 172.21.15.184:0/2139776121 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff23c0095f0 con 0x7ff2480f4860 2024-02-19T22:13:14.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.852+0000 7ff24d9ec700 1 -- 172.21.15.184:0/2139776121 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff248045ad0 con 0x7ff2480f4860 2024-02-19T22:13:14.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.852+0000 7ff22f7fe700 1 --2- 172.21.15.184:0/2139776121 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff23002ff10 0x7ff2300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:14.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.852+0000 7ff22f7fe700 1 -- 172.21.15.184:0/2139776121 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff23c040f50 con 0x7ff2480f4860 2024-02-19T22:13:14.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.852+0000 7ff2467fc700 1 --2- 172.21.15.184:0/2139776121 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff23002ff10 0x7ff2300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:14.855 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.853+0000 7ff2467fc700 1 --2- 172.21.15.184:0/2139776121 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff23002ff10 0x7ff2300323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7ff2380096a0 tx=0x7ff238006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:14.858 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:14.857+0000 7ff22f7fe700 1 -- 172.21.15.184:0/2139776121 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff23c014420 con 0x7ff2480f4860 2024-02-19T22:13:15.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:15.061+0000 7ff24d9ec700 1 -- 172.21.15.184:0/2139776121 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff24805f370 con 0x7ff2480f4860 2024-02-19T22:13:15.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:15.062+0000 7ff22f7fe700 1 -- 172.21.15.184:0/2139776121 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff23c01d980 con 0x7ff2480f4860 2024-02-19T22:13:15.063 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:15.063 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:15.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:15.063+0000 7ff24d9ec700 1 -- 172.21.15.184:0/2139776121 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff23002ff10 msgr2=0x7ff2300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:15.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:15.063+0000 7ff24d9ec700 1 --2- 172.21.15.184:0/2139776121 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff23002ff10 0x7ff2300323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7ff2380096a0 tx=0x7ff238006b40).stop 2024-02-19T22:13:15.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:15.063+0000 7ff24d9ec700 1 -- 172.21.15.184:0/2139776121 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2480f4860 msgr2=0x7ff2481069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:15.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:15.063+0000 7ff24d9ec700 1 --2- 172.21.15.184:0/2139776121 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2480f4860 0x7ff2481069c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7ff23c0089f0 tx=0x7ff23c00b040).stop 2024-02-19T22:13:15.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:15.063+0000 7ff24d9ec700 1 -- 172.21.15.184:0/2139776121 shutdown_connections 2024-02-19T22:13:15.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:15.063+0000 7ff24d9ec700 1 --2- 172.21.15.184:0/2139776121 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff23002ff10 0x7ff2300323d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:15.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:15.063+0000 7ff24d9ec700 1 --2- 172.21.15.184:0/2139776121 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff2480f4860 0x7ff2481069c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:15.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:15.063+0000 7ff24d9ec700 1 -- 172.21.15.184:0/2139776121 >> 172.21.15.184:0/2139776121 conn(0x7ff2480f02e0 msgr2=0x7ff2480f0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:15.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:15.063+0000 7ff24d9ec700 1 -- 172.21.15.184:0/2139776121 shutdown_connections 2024-02-19T22:13:15.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:15.063+0000 7ff24d9ec700 1 -- 172.21.15.184:0/2139776121 wait complete. 2024-02-19T22:13:15.066 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:16.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:13.871323+0000 mgr.smithi176.faoogw (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:16.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:15.063029+0000 mon.smithi176 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.184:0/2139776121' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:16.738 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:16.739 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:18.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:17 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:15.871670+0000 mgr.smithi176.faoogw (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:18.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.687+0000 7f7bb74e4700 1 -- 172.21.15.184:0/2319082894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bb00f6510 msgr2=0x7f7bb00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:18.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.687+0000 7f7bb74e4700 1 --2- 172.21.15.184:0/2319082894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bb00f6510 0x7f7bb00f6930 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f7ba0009fb0 tx=0x7f7ba0009b30).stop 2024-02-19T22:13:18.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.687+0000 7f7bb74e4700 1 -- 172.21.15.184:0/2319082894 shutdown_connections 2024-02-19T22:13:18.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.687+0000 7f7bb74e4700 1 --2- 172.21.15.184:0/2319082894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bb00f6510 0x7f7bb00f6930 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:18.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.687+0000 7f7bb74e4700 1 -- 172.21.15.184:0/2319082894 >> 172.21.15.184:0/2319082894 conn(0x7f7bb00f1c70 msgr2=0x7f7bb00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:18.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.687+0000 7f7bb74e4700 1 -- 172.21.15.184:0/2319082894 shutdown_connections 2024-02-19T22:13:18.700 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.687+0000 7f7bb74e4700 1 -- 172.21.15.184:0/2319082894 wait complete. 2024-02-19T22:13:18.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.688+0000 7f7bb74e4700 1 Processor -- start 2024-02-19T22:13:18.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.688+0000 7f7bb74e4700 1 -- start start 2024-02-19T22:13:18.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.688+0000 7f7bb74e4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bb00f6510 0x7f7bb01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:18.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.688+0000 7f7bb74e4700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb0106cd0 con 0x7f7bb00f6510 2024-02-19T22:13:18.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.689+0000 7f7bb5280700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bb00f6510 0x7f7bb01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:18.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.689+0000 7f7bb5280700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bb00f6510 0x7f7bb01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:34748/0 (socket says 172.21.15.184:34748) 2024-02-19T22:13:18.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.689+0000 7f7bb5280700 1 -- 172.21.15.184:0/3089211271 learned_addr learned my addr 172.21.15.184:0/3089211271 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:18.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.689+0000 7f7bb5280700 1 -- 172.21.15.184:0/3089211271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba0005040 con 0x7f7bb00f6510 2024-02-19T22:13:18.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.689+0000 7f7bb5280700 1 --2- 172.21.15.184:0/3089211271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bb00f6510 0x7f7bb01067e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f7ba0005010 tx=0x7f7ba000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:18.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.689+0000 7f7ba5ffb700 1 -- 172.21.15.184:0/3089211271 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ba0009320 con 0x7f7bb00f6510 2024-02-19T22:13:18.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.689+0000 7f7ba5ffb700 1 -- 172.21.15.184:0/3089211271 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ba0013380 con 0x7f7bb00f6510 2024-02-19T22:13:18.701 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.689+0000 7f7bb74e4700 1 -- 172.21.15.184:0/3089211271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb0106ed0 con 0x7f7bb00f6510 2024-02-19T22:13:18.702 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.689+0000 7f7ba5ffb700 1 -- 172.21.15.184:0/3089211271 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ba0014e50 con 0x7f7bb00f6510 2024-02-19T22:13:18.702 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.689+0000 7f7bb74e4700 1 -- 172.21.15.184:0/3089211271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb0107360 con 0x7f7bb00f6510 2024-02-19T22:13:18.702 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.690+0000 7f7ba5ffb700 1 -- 172.21.15.184:0/3089211271 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ba00095f0 con 0x7f7bb00f6510 2024-02-19T22:13:18.702 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.690+0000 7f7bb74e4700 1 -- 172.21.15.184:0/3089211271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb0045ad0 con 0x7f7bb00f6510 2024-02-19T22:13:18.702 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.690+0000 7f7ba5ffb700 1 --2- 172.21.15.184:0/3089211271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7b9c02ff10 0x7f7b9c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:18.702 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.690+0000 7f7ba5ffb700 1 -- 172.21.15.184:0/3089211271 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ba0040fa0 con 0x7f7bb00f6510 2024-02-19T22:13:18.702 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.690+0000 7f7bb4a7f700 1 --2- 172.21.15.184:0/3089211271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7b9c02ff10 0x7f7b9c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:18.702 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.691+0000 7f7bb4a7f700 1 --2- 172.21.15.184:0/3089211271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7b9c02ff10 0x7f7b9c0323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f7bac0096a0 tx=0x7f7bac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:18.702 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.695+0000 7f7ba5ffb700 1 -- 172.21.15.184:0/3089211271 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ba0014420 con 0x7f7bb00f6510 2024-02-19T22:13:18.901 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.900+0000 7f7bb74e4700 1 -- 172.21.15.184:0/3089211271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bb0108210 con 0x7f7bb00f6510 2024-02-19T22:13:18.902 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.900+0000 7f7ba5ffb700 1 -- 172.21.15.184:0/3089211271 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7ba0010230 con 0x7f7bb00f6510 2024-02-19T22:13:18.902 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:18.902 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:18.903 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.902+0000 7f7bb74e4700 1 -- 172.21.15.184:0/3089211271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7b9c02ff10 msgr2=0x7f7b9c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:18.903 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.902+0000 7f7bb74e4700 1 --2- 172.21.15.184:0/3089211271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7b9c02ff10 0x7f7b9c0323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f7bac0096a0 tx=0x7f7bac006b40).stop 2024-02-19T22:13:18.904 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.902+0000 7f7bb74e4700 1 -- 172.21.15.184:0/3089211271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bb00f6510 msgr2=0x7f7bb01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:18.904 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.902+0000 7f7bb74e4700 1 --2- 172.21.15.184:0/3089211271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bb00f6510 0x7f7bb01067e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f7ba0005010 tx=0x7f7ba000b040).stop 2024-02-19T22:13:18.904 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.902+0000 7f7bb74e4700 1 -- 172.21.15.184:0/3089211271 shutdown_connections 2024-02-19T22:13:18.904 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.902+0000 7f7bb74e4700 1 --2- 172.21.15.184:0/3089211271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7b9c02ff10 0x7f7b9c0323d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:18.904 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.902+0000 7f7bb74e4700 1 --2- 172.21.15.184:0/3089211271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bb00f6510 0x7f7bb01067e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:18.904 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.902+0000 7f7bb74e4700 1 -- 172.21.15.184:0/3089211271 >> 172.21.15.184:0/3089211271 conn(0x7f7bb00f1c70 msgr2=0x7f7bb00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:18.904 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.902+0000 7f7bb74e4700 1 -- 172.21.15.184:0/3089211271 shutdown_connections 2024-02-19T22:13:18.904 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:18.902+0000 7f7bb74e4700 1 -- 172.21.15.184:0/3089211271 wait complete. 2024-02-19T22:13:18.904 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:20.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:17.871927+0000 mgr.smithi176.faoogw (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:20.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:18.901723+0000 mon.smithi176 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.184:0/3089211271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:20.546 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:20.547 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:22.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:19.872244+0000 mgr.smithi176.faoogw (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:22.337 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.335+0000 7fa410b95700 1 -- 172.21.15.184:0/3510684267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa40c0f5330 msgr2=0x7fa40c0f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:22.338 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.335+0000 7fa410b95700 1 --2- 172.21.15.184:0/3510684267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa40c0f5330 0x7fa40c0f5750 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fa3fc009fb0 tx=0x7fa3fc009b30).stop 2024-02-19T22:13:22.338 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.335+0000 7fa410b95700 1 -- 172.21.15.184:0/3510684267 shutdown_connections 2024-02-19T22:13:22.338 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.335+0000 7fa410b95700 1 --2- 172.21.15.184:0/3510684267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa40c0f5330 0x7fa40c0f5750 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:22.338 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.335+0000 7fa410b95700 1 -- 172.21.15.184:0/3510684267 >> 172.21.15.184:0/3510684267 conn(0x7fa40c0f0a90 msgr2=0x7fa40c0f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:22.338 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.335+0000 7fa410b95700 1 -- 172.21.15.184:0/3510684267 shutdown_connections 2024-02-19T22:13:22.338 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.335+0000 7fa410b95700 1 -- 172.21.15.184:0/3510684267 wait complete. 2024-02-19T22:13:22.338 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.335+0000 7fa410b95700 1 Processor -- start 2024-02-19T22:13:22.338 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.335+0000 7fa410b95700 1 -- start start 2024-02-19T22:13:22.338 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.335+0000 7fa410b95700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa40c0f5330 0x7fa40c0fd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:22.338 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.335+0000 7fa410b95700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40c0fb900 con 0x7fa40c0f5330 2024-02-19T22:13:22.339 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.336+0000 7fa40a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa40c0f5330 0x7fa40c0fd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:22.339 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.336+0000 7fa40a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa40c0f5330 0x7fa40c0fd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:34762/0 (socket says 172.21.15.184:34762) 2024-02-19T22:13:22.339 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.336+0000 7fa40a59c700 1 -- 172.21.15.184:0/3059633968 learned_addr learned my addr 172.21.15.184:0/3059633968 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:22.339 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.336+0000 7fa40a59c700 1 -- 172.21.15.184:0/3059633968 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc005040 con 0x7fa40c0f5330 2024-02-19T22:13:22.339 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.336+0000 7fa40a59c700 1 --2- 172.21.15.184:0/3059633968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa40c0f5330 0x7fa40c0fd480 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fa3fc005010 tx=0x7fa3fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:22.339 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.336+0000 7fa3faffd700 1 -- 172.21.15.184:0/3059633968 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3fc009320 con 0x7fa40c0f5330 2024-02-19T22:13:22.339 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.336+0000 7fa3faffd700 1 -- 172.21.15.184:0/3059633968 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3fc013380 con 0x7fa40c0f5330 2024-02-19T22:13:22.339 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.336+0000 7fa3faffd700 1 -- 172.21.15.184:0/3059633968 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3fc014e50 con 0x7fa40c0f5330 2024-02-19T22:13:22.339 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.336+0000 7fa410b95700 1 -- 172.21.15.184:0/3059633968 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40c0fbaa0 con 0x7fa40c0f5330 2024-02-19T22:13:22.339 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.336+0000 7fa410b95700 1 -- 172.21.15.184:0/3059633968 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40c0fbf30 con 0x7fa40c0f5330 2024-02-19T22:13:22.340 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.337+0000 7fa3faffd700 1 -- 172.21.15.184:0/3059633968 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3fc0095f0 con 0x7fa40c0f5330 2024-02-19T22:13:22.340 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.337+0000 7fa410b95700 1 -- 172.21.15.184:0/3059633968 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa40c045ad0 con 0x7fa40c0f5330 2024-02-19T22:13:22.340 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.337+0000 7fa3faffd700 1 --2- 172.21.15.184:0/3059633968 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa3f402ff10 0x7fa3f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:22.340 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.337+0000 7fa3faffd700 1 -- 172.21.15.184:0/3059633968 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3fc040f70 con 0x7fa40c0f5330 2024-02-19T22:13:22.340 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.337+0000 7fa409d9b700 1 --2- 172.21.15.184:0/3059633968 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa3f402ff10 0x7fa3f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:22.340 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.338+0000 7fa409d9b700 1 --2- 172.21.15.184:0/3059633968 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa3f402ff10 0x7fa3f40323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fa4000096a0 tx=0x7fa400006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:22.343 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.342+0000 7fa3faffd700 1 -- 172.21.15.184:0/3059633968 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3fc014420 con 0x7fa40c0f5330 2024-02-19T22:13:22.548 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.546+0000 7fa410b95700 1 -- 172.21.15.184:0/3059633968 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa40c05ea50 con 0x7fa40c0f5330 2024-02-19T22:13:22.548 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.547+0000 7fa3faffd700 1 -- 172.21.15.184:0/3059633968 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa3fc01d980 con 0x7fa40c0f5330 2024-02-19T22:13:22.549 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:22.549 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:22.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.549+0000 7fa410b95700 1 -- 172.21.15.184:0/3059633968 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa3f402ff10 msgr2=0x7fa3f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:22.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.549+0000 7fa410b95700 1 --2- 172.21.15.184:0/3059633968 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa3f402ff10 0x7fa3f40323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fa4000096a0 tx=0x7fa400006b40).stop 2024-02-19T22:13:22.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.549+0000 7fa410b95700 1 -- 172.21.15.184:0/3059633968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa40c0f5330 msgr2=0x7fa40c0fd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:22.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.549+0000 7fa410b95700 1 --2- 172.21.15.184:0/3059633968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa40c0f5330 0x7fa40c0fd480 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fa3fc005010 tx=0x7fa3fc00b040).stop 2024-02-19T22:13:22.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.549+0000 7fa410b95700 1 -- 172.21.15.184:0/3059633968 shutdown_connections 2024-02-19T22:13:22.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.549+0000 7fa410b95700 1 --2- 172.21.15.184:0/3059633968 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa3f402ff10 0x7fa3f40323d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:22.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.549+0000 7fa410b95700 1 --2- 172.21.15.184:0/3059633968 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa40c0f5330 0x7fa40c0fd480 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:22.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.549+0000 7fa410b95700 1 -- 172.21.15.184:0/3059633968 >> 172.21.15.184:0/3059633968 conn(0x7fa40c0f0a90 msgr2=0x7fa40c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:22.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.549+0000 7fa410b95700 1 -- 172.21.15.184:0/3059633968 shutdown_connections 2024-02-19T22:13:22.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:22.549+0000 7fa410b95700 1 -- 172.21.15.184:0/3059633968 wait complete. 2024-02-19T22:13:22.551 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:23.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:22 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:22.548482+0000 mon.smithi176 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.184:0/3059633968' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:24.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:21.872619+0000 mgr.smithi176.faoogw (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:24.222 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:24.223 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:25.979 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.976+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/2411584466 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8c40f51a0 msgr2=0x7fd8c40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:25.979 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.976+0000 7fd8c8e2c700 1 --2- 172.21.15.184:0/2411584466 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8c40f51a0 0x7fd8c40f55c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fd8b4009fb0 tx=0x7fd8b4009b30).stop 2024-02-19T22:13:25.979 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.976+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/2411584466 shutdown_connections 2024-02-19T22:13:25.979 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.976+0000 7fd8c8e2c700 1 --2- 172.21.15.184:0/2411584466 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8c40f51a0 0x7fd8c40f55c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:25.979 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.976+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/2411584466 >> 172.21.15.184:0/2411584466 conn(0x7fd8c40f09a0 msgr2=0x7fd8c40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:25.979 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.976+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/2411584466 shutdown_connections 2024-02-19T22:13:25.979 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.976+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/2411584466 wait complete. 2024-02-19T22:13:25.980 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.976+0000 7fd8c8e2c700 1 Processor -- start 2024-02-19T22:13:25.980 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.976+0000 7fd8c8e2c700 1 -- start start 2024-02-19T22:13:25.980 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.976+0000 7fd8c8e2c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8c40f51a0 0x7fd8c41065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:25.980 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.976+0000 7fd8c8e2c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c4106ac0 con 0x7fd8c40f51a0 2024-02-19T22:13:25.980 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.977+0000 7fd8c37fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8c40f51a0 0x7fd8c41065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:25.980 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.977+0000 7fd8c37fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8c40f51a0 0x7fd8c41065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:34782/0 (socket says 172.21.15.184:34782) 2024-02-19T22:13:25.980 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.977+0000 7fd8c37fe700 1 -- 172.21.15.184:0/1978383302 learned_addr learned my addr 172.21.15.184:0/1978383302 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:25.980 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.977+0000 7fd8c37fe700 1 -- 172.21.15.184:0/1978383302 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b4005040 con 0x7fd8c40f51a0 2024-02-19T22:13:25.980 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.977+0000 7fd8c37fe700 1 --2- 172.21.15.184:0/1978383302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8c40f51a0 0x7fd8c41065d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fd8b40089f0 tx=0x7fd8b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:25.980 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.977+0000 7fd8abfff700 1 -- 172.21.15.184:0/1978383302 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b4009320 con 0x7fd8c40f51a0 2024-02-19T22:13:25.980 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.977+0000 7fd8abfff700 1 -- 172.21.15.184:0/1978383302 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8b4013380 con 0x7fd8c40f51a0 2024-02-19T22:13:25.980 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.977+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/1978383302 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c4106cc0 con 0x7fd8c40f51a0 2024-02-19T22:13:25.981 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.977+0000 7fd8abfff700 1 -- 172.21.15.184:0/1978383302 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b4014e20 con 0x7fd8c40f51a0 2024-02-19T22:13:25.981 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.977+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/1978383302 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c4107150 con 0x7fd8c40f51a0 2024-02-19T22:13:25.981 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.978+0000 7fd8abfff700 1 -- 172.21.15.184:0/1978383302 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8b4014420 con 0x7fd8c40f51a0 2024-02-19T22:13:25.981 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.978+0000 7fd8abfff700 1 --2- 172.21.15.184:0/1978383302 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd8ac02ff10 0x7fd8ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:25.981 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.978+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/1978383302 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c4045ad0 con 0x7fd8c40f51a0 2024-02-19T22:13:25.982 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.978+0000 7fd8abfff700 1 -- 172.21.15.184:0/1978383302 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd8b4040ff0 con 0x7fd8c40f51a0 2024-02-19T22:13:25.982 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.978+0000 7fd8c2ffd700 1 --2- 172.21.15.184:0/1978383302 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd8ac02ff10 0x7fd8ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:25.982 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.979+0000 7fd8c2ffd700 1 --2- 172.21.15.184:0/1978383302 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd8ac02ff10 0x7fd8ac0323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fd8b80096a0 tx=0x7fd8b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:25.984 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:25.983+0000 7fd8abfff700 1 -- 172.21.15.184:0/1978383302 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd8b40095f0 con 0x7fd8c40f51a0 2024-02-19T22:13:26.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:23.872841+0000 mgr.smithi176.faoogw (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:26.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:26.186+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/1978383302 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8c4108000 con 0x7fd8c40f51a0 2024-02-19T22:13:26.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:26.187+0000 7fd8abfff700 1 -- 172.21.15.184:0/1978383302 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd8b401db70 con 0x7fd8c40f51a0 2024-02-19T22:13:26.189 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:26.189 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:26.190 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:26.189+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/1978383302 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd8ac02ff10 msgr2=0x7fd8ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:26.190 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:26.189+0000 7fd8c8e2c700 1 --2- 172.21.15.184:0/1978383302 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd8ac02ff10 0x7fd8ac0323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fd8b80096a0 tx=0x7fd8b8006b40).stop 2024-02-19T22:13:26.190 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:26.189+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/1978383302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8c40f51a0 msgr2=0x7fd8c41065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:26.190 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:26.189+0000 7fd8c8e2c700 1 --2- 172.21.15.184:0/1978383302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8c40f51a0 0x7fd8c41065d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fd8b40089f0 tx=0x7fd8b400b040).stop 2024-02-19T22:13:26.190 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:26.189+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/1978383302 shutdown_connections 2024-02-19T22:13:26.191 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:26.189+0000 7fd8c8e2c700 1 --2- 172.21.15.184:0/1978383302 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd8ac02ff10 0x7fd8ac0323d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:26.191 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:26.189+0000 7fd8c8e2c700 1 --2- 172.21.15.184:0/1978383302 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd8c40f51a0 0x7fd8c41065d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:26.191 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:26.189+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/1978383302 >> 172.21.15.184:0/1978383302 conn(0x7fd8c40f09a0 msgr2=0x7fd8c405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:26.191 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:26.189+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/1978383302 shutdown_connections 2024-02-19T22:13:26.191 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:26.189+0000 7fd8c8e2c700 1 -- 172.21.15.184:0/1978383302 wait complete. 2024-02-19T22:13:26.191 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:27.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:26 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:26.188403+0000 mon.smithi176 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.184:0/1978383302' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:27.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:27.770 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:28.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:25.873162+0000 mgr.smithi176.faoogw (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:29.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.624+0000 7fa2a7418700 1 -- 172.21.15.184:0/2959872232 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2a00f78e0 msgr2=0x7fa2a00f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:29.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.624+0000 7fa2a7418700 1 --2- 172.21.15.184:0/2959872232 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2a00f78e0 0x7fa2a00f7d00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fa29c009fb0 tx=0x7fa29c009b30).stop 2024-02-19T22:13:29.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.625+0000 7fa2a7418700 1 -- 172.21.15.184:0/2959872232 shutdown_connections 2024-02-19T22:13:29.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.625+0000 7fa2a7418700 1 --2- 172.21.15.184:0/2959872232 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2a00f78e0 0x7fa2a00f7d00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:29.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.625+0000 7fa2a7418700 1 -- 172.21.15.184:0/2959872232 >> 172.21.15.184:0/2959872232 conn(0x7fa2a00f0120 msgr2=0x7fa2a00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:29.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.625+0000 7fa2a7418700 1 -- 172.21.15.184:0/2959872232 shutdown_connections 2024-02-19T22:13:29.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.625+0000 7fa2a7418700 1 -- 172.21.15.184:0/2959872232 wait complete. 2024-02-19T22:13:29.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.625+0000 7fa2a7418700 1 Processor -- start 2024-02-19T22:13:29.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.625+0000 7fa2a7418700 1 -- start start 2024-02-19T22:13:29.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.625+0000 7fa2a7418700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2a00f78e0 0x7fa2a00f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:29.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.625+0000 7fa2a7418700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a00f5630 con 0x7fa2a00f78e0 2024-02-19T22:13:29.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.626+0000 7fa2a6416700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2a00f78e0 0x7fa2a00f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:29.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.626+0000 7fa2a6416700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2a00f78e0 0x7fa2a00f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:47480/0 (socket says 172.21.15.184:47480) 2024-02-19T22:13:29.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.626+0000 7fa2a6416700 1 -- 172.21.15.184:0/3301389982 learned_addr learned my addr 172.21.15.184:0/3301389982 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:29.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.626+0000 7fa2a6416700 1 -- 172.21.15.184:0/3301389982 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c005040 con 0x7fa2a00f78e0 2024-02-19T22:13:29.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.626+0000 7fa2a6416700 1 --2- 172.21.15.184:0/3301389982 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2a00f78e0 0x7fa2a00f71b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fa29c0089f0 tx=0x7fa29c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:29.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.626+0000 7fa296ffd700 1 -- 172.21.15.184:0/3301389982 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa29c009320 con 0x7fa2a00f78e0 2024-02-19T22:13:29.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.626+0000 7fa296ffd700 1 -- 172.21.15.184:0/3301389982 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa29c013380 con 0x7fa2a00f78e0 2024-02-19T22:13:29.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.626+0000 7fa2a7418700 1 -- 172.21.15.184:0/3301389982 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a00f57d0 con 0x7fa2a00f78e0 2024-02-19T22:13:29.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.626+0000 7fa296ffd700 1 -- 172.21.15.184:0/3301389982 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa29c014e20 con 0x7fa2a00f78e0 2024-02-19T22:13:29.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.626+0000 7fa2a7418700 1 -- 172.21.15.184:0/3301389982 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a00f5c60 con 0x7fa2a00f78e0 2024-02-19T22:13:29.630 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.627+0000 7fa296ffd700 1 -- 172.21.15.184:0/3301389982 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa29c0095f0 con 0x7fa2a00f78e0 2024-02-19T22:13:29.630 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.627+0000 7fa2a7418700 1 -- 172.21.15.184:0/3301389982 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a0059100 con 0x7fa2a00f78e0 2024-02-19T22:13:29.630 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.627+0000 7fa296ffd700 1 --2- 172.21.15.184:0/3301389982 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa28c02ff10 0x7fa28c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:29.630 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.627+0000 7fa296ffd700 1 -- 172.21.15.184:0/3301389982 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa29c040ed0 con 0x7fa2a00f78e0 2024-02-19T22:13:29.630 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.628+0000 7fa2a5c15700 1 --2- 172.21.15.184:0/3301389982 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa28c02ff10 0x7fa28c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:29.630 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.628+0000 7fa2a5c15700 1 --2- 172.21.15.184:0/3301389982 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa28c02ff10 0x7fa28c0323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fa2900096a0 tx=0x7fa290006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:29.633 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.632+0000 7fa296ffd700 1 -- 172.21.15.184:0/3301389982 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa29c014420 con 0x7fa2a00f78e0 2024-02-19T22:13:29.836 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.835+0000 7fa2a7418700 1 -- 172.21.15.184:0/3301389982 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2a00fbe20 con 0x7fa2a00f78e0 2024-02-19T22:13:29.837 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.835+0000 7fa296ffd700 1 -- 172.21.15.184:0/3301389982 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa29c01d980 con 0x7fa2a00f78e0 2024-02-19T22:13:29.837 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:29.837 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:29.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.837+0000 7fa2a7418700 1 -- 172.21.15.184:0/3301389982 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa28c02ff10 msgr2=0x7fa28c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:29.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.837+0000 7fa2a7418700 1 --2- 172.21.15.184:0/3301389982 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa28c02ff10 0x7fa28c0323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fa2900096a0 tx=0x7fa290006b40).stop 2024-02-19T22:13:29.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.837+0000 7fa2a7418700 1 -- 172.21.15.184:0/3301389982 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2a00f78e0 msgr2=0x7fa2a00f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:29.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.837+0000 7fa2a7418700 1 --2- 172.21.15.184:0/3301389982 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2a00f78e0 0x7fa2a00f71b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fa29c0089f0 tx=0x7fa29c00b040).stop 2024-02-19T22:13:29.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.837+0000 7fa2a7418700 1 -- 172.21.15.184:0/3301389982 shutdown_connections 2024-02-19T22:13:29.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.837+0000 7fa2a7418700 1 --2- 172.21.15.184:0/3301389982 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa28c02ff10 0x7fa28c0323d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:29.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.837+0000 7fa2a7418700 1 --2- 172.21.15.184:0/3301389982 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2a00f78e0 0x7fa2a00f71b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:29.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.837+0000 7fa2a7418700 1 -- 172.21.15.184:0/3301389982 >> 172.21.15.184:0/3301389982 conn(0x7fa2a00f0120 msgr2=0x7fa2a00f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:29.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.837+0000 7fa2a7418700 1 -- 172.21.15.184:0/3301389982 shutdown_connections 2024-02-19T22:13:29.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:29.837+0000 7fa2a7418700 1 -- 172.21.15.184:0/3301389982 wait complete. 2024-02-19T22:13:29.840 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:30.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:29 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:27.873522+0000 mgr.smithi176.faoogw (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:31.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:30 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:29.836799+0000 mon.smithi176 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.184:0/3301389982' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:31.533 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:31.533 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:32.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:29.873885+0000 mgr.smithi176.faoogw (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:33.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.416+0000 7f392a921700 1 -- 172.21.15.184:0/3728051458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39240f51a0 msgr2=0x7f39240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:33.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.416+0000 7f392a921700 1 --2- 172.21.15.184:0/3728051458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39240f51a0 0x7f39240f55c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f3920009fb0 tx=0x7f3920009b30).stop 2024-02-19T22:13:33.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.417+0000 7f392a921700 1 -- 172.21.15.184:0/3728051458 shutdown_connections 2024-02-19T22:13:33.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.417+0000 7f392a921700 1 --2- 172.21.15.184:0/3728051458 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39240f51a0 0x7f39240f55c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:33.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.417+0000 7f392a921700 1 -- 172.21.15.184:0/3728051458 >> 172.21.15.184:0/3728051458 conn(0x7f39240f09a0 msgr2=0x7f39240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:33.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.417+0000 7f392a921700 1 -- 172.21.15.184:0/3728051458 shutdown_connections 2024-02-19T22:13:33.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.417+0000 7f392a921700 1 -- 172.21.15.184:0/3728051458 wait complete. 2024-02-19T22:13:33.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.417+0000 7f392a921700 1 Processor -- start 2024-02-19T22:13:33.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.417+0000 7f392a921700 1 -- start start 2024-02-19T22:13:33.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.417+0000 7f392a921700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39240f51a0 0x7f39240f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:33.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.417+0000 7f392a921700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39240f13a0 con 0x7f39240f51a0 2024-02-19T22:13:33.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.417+0000 7f392991f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39240f51a0 0x7f39240f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:33.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.417+0000 7f392991f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39240f51a0 0x7f39240f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:47500/0 (socket says 172.21.15.184:47500) 2024-02-19T22:13:33.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.417+0000 7f392991f700 1 -- 172.21.15.184:0/2178599654 learned_addr learned my addr 172.21.15.184:0/2178599654 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:33.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.418+0000 7f392991f700 1 -- 172.21.15.184:0/2178599654 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3920005040 con 0x7f39240f51a0 2024-02-19T22:13:33.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.418+0000 7f392991f700 1 --2- 172.21.15.184:0/2178599654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39240f51a0 0x7f39240f2f20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f3920009830 tx=0x7f392000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:33.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.418+0000 7f39167fc700 1 -- 172.21.15.184:0/2178599654 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3920009320 con 0x7f39240f51a0 2024-02-19T22:13:33.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.418+0000 7f392a921700 1 -- 172.21.15.184:0/2178599654 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39240f1540 con 0x7f39240f51a0 2024-02-19T22:13:33.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.418+0000 7f39167fc700 1 -- 172.21.15.184:0/2178599654 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3920013380 con 0x7f39240f51a0 2024-02-19T22:13:33.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.418+0000 7f39167fc700 1 -- 172.21.15.184:0/2178599654 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3920014420 con 0x7f39240f51a0 2024-02-19T22:13:33.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.418+0000 7f392a921700 1 -- 172.21.15.184:0/2178599654 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39240f19d0 con 0x7f39240f51a0 2024-02-19T22:13:33.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.419+0000 7f39167fc700 1 -- 172.21.15.184:0/2178599654 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f392001e420 con 0x7f39240f51a0 2024-02-19T22:13:33.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.419+0000 7f392a921700 1 -- 172.21.15.184:0/2178599654 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3924059100 con 0x7f39240f51a0 2024-02-19T22:13:33.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.419+0000 7f39167fc700 1 --2- 172.21.15.184:0/2178599654 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f391002ff60 0x7f3910032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:33.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.419+0000 7f39167fc700 1 -- 172.21.15.184:0/2178599654 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3920040e30 con 0x7f39240f51a0 2024-02-19T22:13:33.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.419+0000 7f392911e700 1 --2- 172.21.15.184:0/2178599654 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f391002ff60 0x7f3910032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:33.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.420+0000 7f392911e700 1 --2- 172.21.15.184:0/2178599654 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f391002ff60 0x7f3910032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f39180096a0 tx=0x7f3918006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:33.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.424+0000 7f39167fc700 1 -- 172.21.15.184:0/2178599654 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f392001cb10 con 0x7f39240f51a0 2024-02-19T22:13:33.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.628+0000 7f392a921700 1 -- 172.21.15.184:0/2178599654 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3924045ad0 con 0x7f39240f51a0 2024-02-19T22:13:33.630 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.628+0000 7f39167fc700 1 -- 172.21.15.184:0/2178599654 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3920004490 con 0x7f39240f51a0 2024-02-19T22:13:33.630 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:33.630 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:33.631 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.630+0000 7f392a921700 1 -- 172.21.15.184:0/2178599654 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f391002ff60 msgr2=0x7f3910032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:33.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.630+0000 7f392a921700 1 --2- 172.21.15.184:0/2178599654 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f391002ff60 0x7f3910032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f39180096a0 tx=0x7f3918006b40).stop 2024-02-19T22:13:33.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.630+0000 7f392a921700 1 -- 172.21.15.184:0/2178599654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39240f51a0 msgr2=0x7f39240f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:33.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.630+0000 7f392a921700 1 --2- 172.21.15.184:0/2178599654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39240f51a0 0x7f39240f2f20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f3920009830 tx=0x7f392000b040).stop 2024-02-19T22:13:33.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.630+0000 7f392a921700 1 -- 172.21.15.184:0/2178599654 shutdown_connections 2024-02-19T22:13:33.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.630+0000 7f392a921700 1 --2- 172.21.15.184:0/2178599654 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f391002ff60 0x7f3910032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:33.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.630+0000 7f392a921700 1 --2- 172.21.15.184:0/2178599654 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39240f51a0 0x7f39240f2f20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:33.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.630+0000 7f392a921700 1 -- 172.21.15.184:0/2178599654 >> 172.21.15.184:0/2178599654 conn(0x7f39240f09a0 msgr2=0x7f392405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:33.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.630+0000 7f392a921700 1 -- 172.21.15.184:0/2178599654 shutdown_connections 2024-02-19T22:13:33.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:33.630+0000 7f392a921700 1 -- 172.21.15.184:0/2178599654 wait complete. 2024-02-19T22:13:33.633 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:34.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:31.874224+0000 mgr.smithi176.faoogw (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:34.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:33.629779+0000 mon.smithi176 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.184:0/2178599654' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:35.312 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:35.312 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:36.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:33.874609+0000 mgr.smithi176.faoogw (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:37.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.082+0000 7f7135161700 1 -- 172.21.15.184:0/3390273403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71300f5410 msgr2=0x7f71300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:37.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.082+0000 7f7135161700 1 --2- 172.21.15.184:0/3390273403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71300f5410 0x7f71300f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f7120009fb0 tx=0x7f7120009b30).stop 2024-02-19T22:13:37.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.082+0000 7f7135161700 1 -- 172.21.15.184:0/3390273403 shutdown_connections 2024-02-19T22:13:37.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.082+0000 7f7135161700 1 --2- 172.21.15.184:0/3390273403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71300f5410 0x7f71300f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:37.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.082+0000 7f7135161700 1 -- 172.21.15.184:0/3390273403 >> 172.21.15.184:0/3390273403 conn(0x7f71300f0b70 msgr2=0x7f71300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:37.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.082+0000 7f7135161700 1 -- 172.21.15.184:0/3390273403 shutdown_connections 2024-02-19T22:13:37.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.082+0000 7f7135161700 1 -- 172.21.15.184:0/3390273403 wait complete. 2024-02-19T22:13:37.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.083+0000 7f7135161700 1 Processor -- start 2024-02-19T22:13:37.086 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.083+0000 7f7135161700 1 -- start start 2024-02-19T22:13:37.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.083+0000 7f7135161700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71300f5410 0x7f71301094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:37.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.083+0000 7f7135161700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7130060240 con 0x7f71300f5410 2024-02-19T22:13:37.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.083+0000 7f712ed9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71300f5410 0x7f71301094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:37.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.083+0000 7f712ed9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71300f5410 0x7f71301094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:57714/0 (socket says 172.21.15.184:57714) 2024-02-19T22:13:37.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.084+0000 7f712ed9d700 1 -- 172.21.15.184:0/2105920056 learned_addr learned my addr 172.21.15.184:0/2105920056 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:37.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.084+0000 7f712ed9d700 1 -- 172.21.15.184:0/2105920056 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7120005040 con 0x7f71300f5410 2024-02-19T22:13:37.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.084+0000 7f712ed9d700 1 --2- 172.21.15.184:0/2105920056 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71300f5410 0x7f71301094e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f7120005010 tx=0x7f712000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:37.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.084+0000 7f71277fe700 1 -- 172.21.15.184:0/2105920056 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7120009320 con 0x7f71300f5410 2024-02-19T22:13:37.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.084+0000 7f7135161700 1 -- 172.21.15.184:0/2105920056 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7130060440 con 0x7f71300f5410 2024-02-19T22:13:37.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.084+0000 7f71277fe700 1 -- 172.21.15.184:0/2105920056 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7120013380 con 0x7f71300f5410 2024-02-19T22:13:37.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.084+0000 7f71277fe700 1 -- 172.21.15.184:0/2105920056 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7120014e50 con 0x7f71300f5410 2024-02-19T22:13:37.087 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.084+0000 7f7135161700 1 -- 172.21.15.184:0/2105920056 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7130109b90 con 0x7f71300f5410 2024-02-19T22:13:37.088 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.085+0000 7f71277fe700 1 -- 172.21.15.184:0/2105920056 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71200095f0 con 0x7f71300f5410 2024-02-19T22:13:37.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.085+0000 7f7135161700 1 -- 172.21.15.184:0/2105920056 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7130045ad0 con 0x7f71300f5410 2024-02-19T22:13:37.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.085+0000 7f71277fe700 1 --2- 172.21.15.184:0/2105920056 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f711c02ff10 0x7f711c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:37.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.085+0000 7f71277fe700 1 -- 172.21.15.184:0/2105920056 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7120040e90 con 0x7f71300f5410 2024-02-19T22:13:37.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.085+0000 7f712e59c700 1 --2- 172.21.15.184:0/2105920056 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f711c02ff10 0x7f711c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:37.089 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.086+0000 7f712e59c700 1 --2- 172.21.15.184:0/2105920056 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f711c02ff10 0x7f711c0323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f71180096a0 tx=0x7f7118006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:37.091 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.090+0000 7f71277fe700 1 -- 172.21.15.184:0/2105920056 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7120014420 con 0x7f71300f5410 2024-02-19T22:13:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:35.970716+0000 mon.smithi176 (mon.0) 196 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:13:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:35.973144+0000 mon.smithi176 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:13:37.295 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.294+0000 7f7135161700 1 -- 172.21.15.184:0/2105920056 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f713010a3b0 con 0x7f71300f5410 2024-02-19T22:13:37.296 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.295+0000 7f71277fe700 1 -- 172.21.15.184:0/2105920056 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7120010230 con 0x7f71300f5410 2024-02-19T22:13:37.296 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:37.296 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:37.297 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.297+0000 7f7135161700 1 -- 172.21.15.184:0/2105920056 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f711c02ff10 msgr2=0x7f711c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:37.297 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.297+0000 7f7135161700 1 --2- 172.21.15.184:0/2105920056 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f711c02ff10 0x7f711c0323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f71180096a0 tx=0x7f7118006b40).stop 2024-02-19T22:13:37.298 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.297+0000 7f7135161700 1 -- 172.21.15.184:0/2105920056 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71300f5410 msgr2=0x7f71301094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:37.298 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.297+0000 7f7135161700 1 --2- 172.21.15.184:0/2105920056 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71300f5410 0x7f71301094e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f7120005010 tx=0x7f712000b040).stop 2024-02-19T22:13:37.298 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.297+0000 7f7135161700 1 -- 172.21.15.184:0/2105920056 shutdown_connections 2024-02-19T22:13:37.298 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.297+0000 7f7135161700 1 --2- 172.21.15.184:0/2105920056 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f711c02ff10 0x7f711c0323d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:37.298 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.297+0000 7f7135161700 1 --2- 172.21.15.184:0/2105920056 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71300f5410 0x7f71301094e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:37.298 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.297+0000 7f7135161700 1 -- 172.21.15.184:0/2105920056 >> 172.21.15.184:0/2105920056 conn(0x7f71300f0b70 msgr2=0x7f71300f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:37.298 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.297+0000 7f7135161700 1 -- 172.21.15.184:0/2105920056 shutdown_connections 2024-02-19T22:13:37.298 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:37.297+0000 7f7135161700 1 -- 172.21.15.184:0/2105920056 wait complete. 2024-02-19T22:13:37.298 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:38.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:35.874828+0000 mgr.smithi176.faoogw (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:38.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:37.295462+0000 mon.smithi176 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.184:0/2105920056' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:38.922 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:38.923 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:40.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:37.875058+0000 mgr.smithi176.faoogw (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:40.746 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.744+0000 7f93d509e700 1 -- 172.21.15.184:0/2001651204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d00f6510 msgr2=0x7f93d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:40.746 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.744+0000 7f93d509e700 1 --2- 172.21.15.184:0/2001651204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d00f6510 0x7f93d00f6930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f93c4009fb0 tx=0x7f93c4009b30).stop 2024-02-19T22:13:40.746 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.744+0000 7f93d509e700 1 -- 172.21.15.184:0/2001651204 shutdown_connections 2024-02-19T22:13:40.746 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.744+0000 7f93d509e700 1 --2- 172.21.15.184:0/2001651204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d00f6510 0x7f93d00f6930 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:40.746 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.744+0000 7f93d509e700 1 -- 172.21.15.184:0/2001651204 >> 172.21.15.184:0/2001651204 conn(0x7f93d00f1c70 msgr2=0x7f93d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:40.747 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.744+0000 7f93d509e700 1 -- 172.21.15.184:0/2001651204 shutdown_connections 2024-02-19T22:13:40.747 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.744+0000 7f93d509e700 1 -- 172.21.15.184:0/2001651204 wait complete. 2024-02-19T22:13:40.747 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.745+0000 7f93d509e700 1 Processor -- start 2024-02-19T22:13:40.747 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.745+0000 7f93d509e700 1 -- start start 2024-02-19T22:13:40.747 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.745+0000 7f93d509e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d00f6510 0x7f93d0108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:40.747 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.745+0000 7f93d509e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d0108ef0 con 0x7f93d00f6510 2024-02-19T22:13:40.747 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.745+0000 7f93ced9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d00f6510 0x7f93d0108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:40.747 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.745+0000 7f93ced9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d00f6510 0x7f93d0108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:57740/0 (socket says 172.21.15.184:57740) 2024-02-19T22:13:40.747 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.745+0000 7f93ced9d700 1 -- 172.21.15.184:0/1355618228 learned_addr learned my addr 172.21.15.184:0/1355618228 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:40.747 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.745+0000 7f93ced9d700 1 -- 172.21.15.184:0/1355618228 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c4005040 con 0x7f93d00f6510 2024-02-19T22:13:40.747 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.745+0000 7f93ced9d700 1 --2- 172.21.15.184:0/1355618228 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d00f6510 0x7f93d0108a00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f93c4005010 tx=0x7f93c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:40.747 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.745+0000 7f93b77fe700 1 -- 172.21.15.184:0/1355618228 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93c4009320 con 0x7f93d00f6510 2024-02-19T22:13:40.748 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.746+0000 7f93b77fe700 1 -- 172.21.15.184:0/1355618228 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93c4013380 con 0x7f93d00f6510 2024-02-19T22:13:40.748 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.746+0000 7f93d509e700 1 -- 172.21.15.184:0/1355618228 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d01090f0 con 0x7f93d00f6510 2024-02-19T22:13:40.748 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.746+0000 7f93b77fe700 1 -- 172.21.15.184:0/1355618228 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93c4014e50 con 0x7f93d00f6510 2024-02-19T22:13:40.748 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.746+0000 7f93d509e700 1 -- 172.21.15.184:0/1355618228 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d0109580 con 0x7f93d00f6510 2024-02-19T22:13:40.749 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.746+0000 7f93b77fe700 1 -- 172.21.15.184:0/1355618228 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93c40095f0 con 0x7f93d00f6510 2024-02-19T22:13:40.749 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.746+0000 7f93d509e700 1 -- 172.21.15.184:0/1355618228 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d0045ad0 con 0x7f93d00f6510 2024-02-19T22:13:40.749 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.747+0000 7f93b77fe700 1 --2- 172.21.15.184:0/1355618228 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f93b802ff10 0x7f93b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:40.749 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.747+0000 7f93b77fe700 1 -- 172.21.15.184:0/1355618228 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f93c4040e90 con 0x7f93d00f6510 2024-02-19T22:13:40.749 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.747+0000 7f93ce59c700 1 --2- 172.21.15.184:0/1355618228 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f93b802ff10 0x7f93b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:40.749 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.747+0000 7f93ce59c700 1 --2- 172.21.15.184:0/1355618228 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f93b802ff10 0x7f93b80323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f93c00096a0 tx=0x7f93c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:40.751 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.751+0000 7f93b77fe700 1 -- 172.21.15.184:0/1355618228 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93c4014420 con 0x7f93d00f6510 2024-02-19T22:13:40.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.956+0000 7f93d509e700 1 -- 172.21.15.184:0/1355618228 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93d010a430 con 0x7f93d00f6510 2024-02-19T22:13:40.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.957+0000 7f93b77fe700 1 -- 172.21.15.184:0/1355618228 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f93c4010230 con 0x7f93d00f6510 2024-02-19T22:13:40.958 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:40.958 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:40.959 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.958+0000 7f93d509e700 1 -- 172.21.15.184:0/1355618228 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f93b802ff10 msgr2=0x7f93b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:40.959 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.958+0000 7f93d509e700 1 --2- 172.21.15.184:0/1355618228 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f93b802ff10 0x7f93b80323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f93c00096a0 tx=0x7f93c0006b40).stop 2024-02-19T22:13:40.959 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.958+0000 7f93d509e700 1 -- 172.21.15.184:0/1355618228 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d00f6510 msgr2=0x7f93d0108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:40.959 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.958+0000 7f93d509e700 1 --2- 172.21.15.184:0/1355618228 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d00f6510 0x7f93d0108a00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f93c4005010 tx=0x7f93c400b040).stop 2024-02-19T22:13:40.959 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.958+0000 7f93d509e700 1 -- 172.21.15.184:0/1355618228 shutdown_connections 2024-02-19T22:13:40.960 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.958+0000 7f93d509e700 1 --2- 172.21.15.184:0/1355618228 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f93b802ff10 0x7f93b80323d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:40.960 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.958+0000 7f93d509e700 1 --2- 172.21.15.184:0/1355618228 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93d00f6510 0x7f93d0108a00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:40.960 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.958+0000 7f93d509e700 1 -- 172.21.15.184:0/1355618228 >> 172.21.15.184:0/1355618228 conn(0x7f93d00f1c70 msgr2=0x7f93d00f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:40.960 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.958+0000 7f93d509e700 1 -- 172.21.15.184:0/1355618228 shutdown_connections 2024-02-19T22:13:40.960 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:40.959+0000 7f93d509e700 1 -- 172.21.15.184:0/1355618228 wait complete. 2024-02-19T22:13:40.960 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:42.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:39.875355+0000 mgr.smithi176.faoogw (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:42.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:40.957253+0000 mon.smithi176 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.184:0/1355618228' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:42.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:42.597 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:44.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:41.875685+0000 mgr.smithi176.faoogw (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:44.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.420+0000 7f84fe96c700 1 -- 172.21.15.184:0/452921974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84f80f5410 msgr2=0x7f84f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:44.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.420+0000 7f84fe96c700 1 --2- 172.21.15.184:0/452921974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84f80f5410 0x7f84f80f5830 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f84ec009fb0 tx=0x7f84ec009b30).stop 2024-02-19T22:13:44.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.420+0000 7f84fe96c700 1 -- 172.21.15.184:0/452921974 shutdown_connections 2024-02-19T22:13:44.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.420+0000 7f84fe96c700 1 --2- 172.21.15.184:0/452921974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84f80f5410 0x7f84f80f5830 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:44.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.420+0000 7f84fe96c700 1 -- 172.21.15.184:0/452921974 >> 172.21.15.184:0/452921974 conn(0x7f84f80f0b70 msgr2=0x7f84f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:44.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.420+0000 7f84fe96c700 1 -- 172.21.15.184:0/452921974 shutdown_connections 2024-02-19T22:13:44.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.420+0000 7f84fe96c700 1 -- 172.21.15.184:0/452921974 wait complete. 2024-02-19T22:13:44.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.420+0000 7f84fe96c700 1 Processor -- start 2024-02-19T22:13:44.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.421+0000 7f84fe96c700 1 -- start start 2024-02-19T22:13:44.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.421+0000 7f84fe96c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84f80f5410 0x7f84f81089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:44.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.421+0000 7f84fe96c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f8108ed0 con 0x7f84f80f5410 2024-02-19T22:13:44.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.421+0000 7f84f7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84f80f5410 0x7f84f81089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:44.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.421+0000 7f84f7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84f80f5410 0x7f84f81089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:57758/0 (socket says 172.21.15.184:57758) 2024-02-19T22:13:44.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.421+0000 7f84f7fff700 1 -- 172.21.15.184:0/1250414921 learned_addr learned my addr 172.21.15.184:0/1250414921 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:44.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.421+0000 7f84f7fff700 1 -- 172.21.15.184:0/1250414921 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ec005040 con 0x7f84f80f5410 2024-02-19T22:13:44.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.421+0000 7f84f7fff700 1 --2- 172.21.15.184:0/1250414921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84f80f5410 0x7f84f81089e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f84ec005010 tx=0x7f84ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:44.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.421+0000 7f84f4ff9700 1 -- 172.21.15.184:0/1250414921 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84ec009320 con 0x7f84f80f5410 2024-02-19T22:13:44.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.421+0000 7f84fe96c700 1 -- 172.21.15.184:0/1250414921 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f81090d0 con 0x7f84f80f5410 2024-02-19T22:13:44.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.421+0000 7f84f4ff9700 1 -- 172.21.15.184:0/1250414921 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84ec013380 con 0x7f84f80f5410 2024-02-19T22:13:44.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.421+0000 7f84f4ff9700 1 -- 172.21.15.184:0/1250414921 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84ec014e50 con 0x7f84f80f5410 2024-02-19T22:13:44.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.422+0000 7f84fe96c700 1 -- 172.21.15.184:0/1250414921 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f8109560 con 0x7f84f80f5410 2024-02-19T22:13:44.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.422+0000 7f84f4ff9700 1 -- 172.21.15.184:0/1250414921 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84ec0263a0 con 0x7f84f80f5410 2024-02-19T22:13:44.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.422+0000 7f84f4ff9700 1 --2- 172.21.15.184:0/1250414921 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f84e002ff00 0x7f84e00323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:44.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.422+0000 7f84f4ff9700 1 -- 172.21.15.184:0/1250414921 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84ec042020 con 0x7f84f80f5410 2024-02-19T22:13:44.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.423+0000 7f84fe96c700 1 -- 172.21.15.184:0/1250414921 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f8059100 con 0x7f84f80f5410 2024-02-19T22:13:44.426 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.423+0000 7f84f77fe700 1 --2- 172.21.15.184:0/1250414921 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f84e002ff00 0x7f84e00323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:44.426 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.423+0000 7f84f77fe700 1 --2- 172.21.15.184:0/1250414921 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f84e002ff00 0x7f84e00323c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f84e80096a0 tx=0x7f84e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:44.428 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.427+0000 7f84f4ff9700 1 -- 172.21.15.184:0/1250414921 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84ec012920 con 0x7f84f80f5410 2024-02-19T22:13:44.633 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.631+0000 7f84fe96c700 1 -- 172.21.15.184:0/1250414921 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84f80f9ec0 con 0x7f84f80f5410 2024-02-19T22:13:44.634 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.632+0000 7f84f4ff9700 1 -- 172.21.15.184:0/1250414921 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f84ec02b020 con 0x7f84f80f5410 2024-02-19T22:13:44.634 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:44.634 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:44.635 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.634+0000 7f84fe96c700 1 -- 172.21.15.184:0/1250414921 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f84e002ff00 msgr2=0x7f84e00323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:44.635 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.634+0000 7f84fe96c700 1 --2- 172.21.15.184:0/1250414921 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f84e002ff00 0x7f84e00323c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f84e80096a0 tx=0x7f84e8006b40).stop 2024-02-19T22:13:44.635 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.634+0000 7f84fe96c700 1 -- 172.21.15.184:0/1250414921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84f80f5410 msgr2=0x7f84f81089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:44.636 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.634+0000 7f84fe96c700 1 --2- 172.21.15.184:0/1250414921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84f80f5410 0x7f84f81089e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f84ec005010 tx=0x7f84ec00b040).stop 2024-02-19T22:13:44.636 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.634+0000 7f84fe96c700 1 -- 172.21.15.184:0/1250414921 shutdown_connections 2024-02-19T22:13:44.636 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.634+0000 7f84fe96c700 1 --2- 172.21.15.184:0/1250414921 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f84e002ff00 0x7f84e00323c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:44.636 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.634+0000 7f84fe96c700 1 --2- 172.21.15.184:0/1250414921 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f84f80f5410 0x7f84f81089e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:44.636 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.634+0000 7f84fe96c700 1 -- 172.21.15.184:0/1250414921 >> 172.21.15.184:0/1250414921 conn(0x7f84f80f0b70 msgr2=0x7f84f80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:44.636 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.634+0000 7f84fe96c700 1 -- 172.21.15.184:0/1250414921 shutdown_connections 2024-02-19T22:13:44.636 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:44.634+0000 7f84fe96c700 1 -- 172.21.15.184:0/1250414921 wait complete. 2024-02-19T22:13:44.636 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:45.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:44 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:44.207481+0000 mon.smithi176 (mon.0) 200 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi176", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T22:13:45.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:44 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:44.210075+0000 mon.smithi176 (mon.0) 201 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi176", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-19T22:13:45.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:44 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:44.210840+0000 mon.smithi176 (mon.0) 202 : audit [DBG] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T22:13:45.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:44 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:44.633403+0000 mon.smithi176 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.184:0/1250414921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:46.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:43.876016+0000 mgr.smithi176.faoogw (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:46.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:13:44.205880+0000 mgr.smithi176.faoogw (mgr.14162) 156 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi176 ... 2024-02-19T22:13:46.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-19T22:13:46.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:13:46.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:09:22Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:13:46.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:10:00Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:13:46.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:10:37Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:13:46.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:13:46.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-19T22:13:46.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:13:46.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:11:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:13:46.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:12:29Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:13:46.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:13:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:13:46.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:13:46.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Traceback (most recent call last): 2024-02-19T22:13:46.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:13:46.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: main() 2024-02-19T22:13:46.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:13:46.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: r = ctx.func(ctx) 2024-02-19T22:13:46.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:13:46.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: return func(ctx) 2024-02-19T22:13:46.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:13:46.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:13:46.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-19T22:13:46.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-19T22:13:46.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:13:46.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: raise RuntimeError('uid/gid not found') 2024-02-19T22:13:46.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: RuntimeError: uid/gid not found 2024-02-19T22:13:46.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Traceback (most recent call last): 2024-02-19T22:13:46.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:13:46.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: yield (conn, connr) 2024-02-19T22:13:46.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:13:46.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: code, '\n'.join(err))) 2024-02-19T22:13:46.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi176 ... 2024-02-19T22:13:46.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-19T22:13:46.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:13:46.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:09:22Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:13:46.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:10:00Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:13:46.140 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:10:37Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:13:46.140 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:13:46.140 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-19T22:13:46.140 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:13:46.140 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:11:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:13:46.140 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:12:29Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:13:46.140 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:13:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:13:46.140 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:13:46.140 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Traceback (most recent call last): 2024-02-19T22:13:46.140 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:13:46.140 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: main() 2024-02-19T22:13:46.140 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:13:46.141 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: r = ctx.func(ctx) 2024-02-19T22:13:46.141 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:13:46.141 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: return func(ctx) 2024-02-19T22:13:46.141 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:13:46.141 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:13:46.141 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-19T22:13:46.141 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-19T22:13:46.141 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:13:46.141 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: raise RuntimeError('uid/gid not found') 2024-02-19T22:13:46.141 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: RuntimeError: uid/gid not found 2024-02-19T22:13:46.141 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:13:44.211416+0000 mgr.smithi176.faoogw (mgr.14162) 157 : cephadm [INF] Deploying daemon crash.smithi176 on smithi176 2024-02-19T22:13:46.265 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:46.265 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:48.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:45.876245+0000 mgr.smithi176.faoogw (mgr.14162) 158 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:48.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:46.649476+0000 mon.smithi176 (mon.0) 204 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:13:48.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:46.812453+0000 mon.smithi176 (mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:13:48.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:46.817108+0000 mon.smithi176 (mon.0) 206 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:13:48.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:46.817535+0000 mon.smithi176 (mon.0) 207 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T22:13:48.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:46.822939+0000 mon.smithi176 (mon.0) 208 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:13:48.195 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.192+0000 7fa31c65e700 1 -- 172.21.15.184:0/2217582804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3140f6480 msgr2=0x7fa3140f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:48.195 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.192+0000 7fa31c65e700 1 --2- 172.21.15.184:0/2217582804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3140f6480 0x7fa3140f68a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fa304009fb0 tx=0x7fa304009b30).stop 2024-02-19T22:13:48.195 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.192+0000 7fa31c65e700 1 -- 172.21.15.184:0/2217582804 shutdown_connections 2024-02-19T22:13:48.195 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.192+0000 7fa31c65e700 1 --2- 172.21.15.184:0/2217582804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3140f6480 0x7fa3140f68a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:48.195 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.192+0000 7fa31c65e700 1 -- 172.21.15.184:0/2217582804 >> 172.21.15.184:0/2217582804 conn(0x7fa3140f1c00 msgr2=0x7fa3140f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:48.195 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.192+0000 7fa31c65e700 1 -- 172.21.15.184:0/2217582804 shutdown_connections 2024-02-19T22:13:48.195 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.192+0000 7fa31c65e700 1 -- 172.21.15.184:0/2217582804 wait complete. 2024-02-19T22:13:48.195 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.193+0000 7fa31c65e700 1 Processor -- start 2024-02-19T22:13:48.195 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.193+0000 7fa31c65e700 1 -- start start 2024-02-19T22:13:48.195 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.193+0000 7fa31c65e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3140f6480 0x7fa31406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:48.195 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.193+0000 7fa31c65e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa314069e20 con 0x7fa3140f6480 2024-02-19T22:13:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.194+0000 7fa31a3fa700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3140f6480 0x7fa31406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.194+0000 7fa31a3fa700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3140f6480 0x7fa31406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:43290/0 (socket says 172.21.15.184:43290) 2024-02-19T22:13:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.194+0000 7fa31a3fa700 1 -- 172.21.15.184:0/1763831334 learned_addr learned my addr 172.21.15.184:0/1763831334 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.194+0000 7fa31a3fa700 1 -- 172.21.15.184:0/1763831334 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa304005040 con 0x7fa3140f6480 2024-02-19T22:13:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.194+0000 7fa31a3fa700 1 --2- 172.21.15.184:0/1763831334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3140f6480 0x7fa31406b9a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fa304005010 tx=0x7fa30400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.194+0000 7fa30affd700 1 -- 172.21.15.184:0/1763831334 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa304009320 con 0x7fa3140f6480 2024-02-19T22:13:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.194+0000 7fa31c65e700 1 -- 172.21.15.184:0/1763831334 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa314069fc0 con 0x7fa3140f6480 2024-02-19T22:13:48.197 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.194+0000 7fa30affd700 1 -- 172.21.15.184:0/1763831334 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa304013380 con 0x7fa3140f6480 2024-02-19T22:13:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.194+0000 7fa30affd700 1 -- 172.21.15.184:0/1763831334 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa304014e50 con 0x7fa3140f6480 2024-02-19T22:13:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.194+0000 7fa31c65e700 1 -- 172.21.15.184:0/1763831334 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31406a450 con 0x7fa3140f6480 2024-02-19T22:13:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.195+0000 7fa30affd700 1 -- 172.21.15.184:0/1763831334 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa304014420 con 0x7fa3140f6480 2024-02-19T22:13:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.195+0000 7fa31c65e700 1 -- 172.21.15.184:0/1763831334 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa314059100 con 0x7fa3140f6480 2024-02-19T22:13:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.195+0000 7fa30affd700 1 --2- 172.21.15.184:0/1763831334 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa30002ff10 0x7fa3000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.195+0000 7fa30affd700 1 -- 172.21.15.184:0/1763831334 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa304040f40 con 0x7fa3140f6480 2024-02-19T22:13:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.195+0000 7fa319bf9700 1 --2- 172.21.15.184:0/1763831334 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa30002ff10 0x7fa3000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:48.198 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.196+0000 7fa319bf9700 1 --2- 172.21.15.184:0/1763831334 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa30002ff10 0x7fa3000323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa3100096a0 tx=0x7fa310006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:48.201 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.200+0000 7fa30affd700 1 -- 172.21.15.184:0/1763831334 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3040095f0 con 0x7fa3140f6480 2024-02-19T22:13:48.405 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.403+0000 7fa31c65e700 1 -- 172.21.15.184:0/1763831334 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3140fc090 con 0x7fa3140f6480 2024-02-19T22:13:48.405 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.404+0000 7fa30affd700 1 -- 172.21.15.184:0/1763831334 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa30401db70 con 0x7fa3140f6480 2024-02-19T22:13:48.405 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:48.406 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:48.407 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.405+0000 7fa31c65e700 1 -- 172.21.15.184:0/1763831334 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa30002ff10 msgr2=0x7fa3000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:48.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.405+0000 7fa31c65e700 1 --2- 172.21.15.184:0/1763831334 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa30002ff10 0x7fa3000323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa3100096a0 tx=0x7fa310006b40).stop 2024-02-19T22:13:48.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.405+0000 7fa31c65e700 1 -- 172.21.15.184:0/1763831334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3140f6480 msgr2=0x7fa31406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:48.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.405+0000 7fa31c65e700 1 --2- 172.21.15.184:0/1763831334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3140f6480 0x7fa31406b9a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fa304005010 tx=0x7fa30400b040).stop 2024-02-19T22:13:48.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.405+0000 7fa31c65e700 1 -- 172.21.15.184:0/1763831334 shutdown_connections 2024-02-19T22:13:48.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.405+0000 7fa31c65e700 1 --2- 172.21.15.184:0/1763831334 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa30002ff10 0x7fa3000323d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:48.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.405+0000 7fa31c65e700 1 --2- 172.21.15.184:0/1763831334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3140f6480 0x7fa31406b9a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:48.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.405+0000 7fa31c65e700 1 -- 172.21.15.184:0/1763831334 >> 172.21.15.184:0/1763831334 conn(0x7fa3140f1c00 msgr2=0x7fa3140f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:48.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.405+0000 7fa31c65e700 1 -- 172.21.15.184:0/1763831334 shutdown_connections 2024-02-19T22:13:48.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:48.405+0000 7fa31c65e700 1 -- 172.21.15.184:0/1763831334 wait complete. 2024-02-19T22:13:48.408 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:49.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:48 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:46.817969+0000 mgr.smithi176.faoogw (mgr.14162) 159 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T22:13:49.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:48 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:13:46.826115+0000 mgr.smithi176.faoogw (mgr.14162) 160 : cephadm [INF] Deploying daemon grafana.smithi176 on smithi176 2024-02-19T22:13:49.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:48 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:48.405192+0000 mon.smithi176 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.184:0/1763831334' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:50.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:49 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:47.876544+0000 mgr.smithi176.faoogw (mgr.14162) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:50.182 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:50.182 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:52.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.068+0000 7f53db3c2700 1 -- 172.21.15.184:0/2171375936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53d4060280 msgr2=0x7f53d40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:52.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.068+0000 7f53db3c2700 1 --2- 172.21.15.184:0/2171375936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53d4060280 0x7f53d40f90f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f53c4009fb0 tx=0x7f53c4009b30).stop 2024-02-19T22:13:52.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.068+0000 7f53db3c2700 1 -- 172.21.15.184:0/2171375936 shutdown_connections 2024-02-19T22:13:52.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.068+0000 7f53db3c2700 1 --2- 172.21.15.184:0/2171375936 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53d4060280 0x7f53d40f90f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:52.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.068+0000 7f53db3c2700 1 -- 172.21.15.184:0/2171375936 >> 172.21.15.184:0/2171375936 conn(0x7f53d40f0b70 msgr2=0x7f53d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:52.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.068+0000 7f53db3c2700 1 -- 172.21.15.184:0/2171375936 shutdown_connections 2024-02-19T22:13:52.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.068+0000 7f53db3c2700 1 -- 172.21.15.184:0/2171375936 wait complete. 2024-02-19T22:13:52.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.069+0000 7f53db3c2700 1 Processor -- start 2024-02-19T22:13:52.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.069+0000 7f53db3c2700 1 -- start start 2024-02-19T22:13:52.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.069+0000 7f53db3c2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53d4060280 0x7f53d4106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:52.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.069+0000 7f53db3c2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d4106c70 con 0x7f53d4060280 2024-02-19T22:13:52.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.069+0000 7f53d915e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53d4060280 0x7f53d4106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:52.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.069+0000 7f53d915e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53d4060280 0x7f53d4106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:43304/0 (socket says 172.21.15.184:43304) 2024-02-19T22:13:52.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.069+0000 7f53d915e700 1 -- 172.21.15.184:0/320720974 learned_addr learned my addr 172.21.15.184:0/320720974 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:52.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.069+0000 7f53d915e700 1 -- 172.21.15.184:0/320720974 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c4005040 con 0x7f53d4060280 2024-02-19T22:13:52.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.069+0000 7f53d915e700 1 --2- 172.21.15.184:0/320720974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53d4060280 0x7f53d4106780 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f53c40089f0 tx=0x7f53c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:52.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.070+0000 7f53c9ffb700 1 -- 172.21.15.184:0/320720974 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53c4009320 con 0x7f53d4060280 2024-02-19T22:13:52.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.070+0000 7f53c9ffb700 1 -- 172.21.15.184:0/320720974 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53c4013380 con 0x7f53d4060280 2024-02-19T22:13:52.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.070+0000 7f53db3c2700 1 -- 172.21.15.184:0/320720974 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d4106e70 con 0x7f53d4060280 2024-02-19T22:13:52.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.070+0000 7f53c9ffb700 1 -- 172.21.15.184:0/320720974 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53c4014e20 con 0x7f53d4060280 2024-02-19T22:13:52.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.070+0000 7f53db3c2700 1 -- 172.21.15.184:0/320720974 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d4107300 con 0x7f53d4060280 2024-02-19T22:13:52.074 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.071+0000 7f53c9ffb700 1 -- 172.21.15.184:0/320720974 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53c4014420 con 0x7f53d4060280 2024-02-19T22:13:52.074 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.071+0000 7f53db3c2700 1 -- 172.21.15.184:0/320720974 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d4045ad0 con 0x7f53d4060280 2024-02-19T22:13:52.074 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.071+0000 7f53c9ffb700 1 --2- 172.21.15.184:0/320720974 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f53c002ff10 0x7f53c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:52.074 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.071+0000 7f53c9ffb700 1 -- 172.21.15.184:0/320720974 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f53c4040ff0 con 0x7f53d4060280 2024-02-19T22:13:52.074 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.071+0000 7f53d895d700 1 --2- 172.21.15.184:0/320720974 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f53c002ff10 0x7f53c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:52.074 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.071+0000 7f53d895d700 1 --2- 172.21.15.184:0/320720974 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f53c002ff10 0x7f53c00323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f53d00096a0 tx=0x7f53d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:52.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.075+0000 7f53c9ffb700 1 -- 172.21.15.184:0/320720974 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53c40095f0 con 0x7f53d4060280 2024-02-19T22:13:52.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:51 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:49.876883+0000 mgr.smithi176.faoogw (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:52.280 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.279+0000 7f53db3c2700 1 -- 172.21.15.184:0/320720974 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53d41081b0 con 0x7f53d4060280 2024-02-19T22:13:52.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.279+0000 7f53c9ffb700 1 -- 172.21.15.184:0/320720974 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f53c401db70 con 0x7f53d4060280 2024-02-19T22:13:52.281 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:52.282 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:52.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.281+0000 7f53db3c2700 1 -- 172.21.15.184:0/320720974 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f53c002ff10 msgr2=0x7f53c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:52.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.281+0000 7f53db3c2700 1 --2- 172.21.15.184:0/320720974 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f53c002ff10 0x7f53c00323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f53d00096a0 tx=0x7f53d0006b40).stop 2024-02-19T22:13:52.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.281+0000 7f53db3c2700 1 -- 172.21.15.184:0/320720974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53d4060280 msgr2=0x7f53d4106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:52.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.281+0000 7f53db3c2700 1 --2- 172.21.15.184:0/320720974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53d4060280 0x7f53d4106780 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f53c40089f0 tx=0x7f53c400b040).stop 2024-02-19T22:13:52.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.281+0000 7f53db3c2700 1 -- 172.21.15.184:0/320720974 shutdown_connections 2024-02-19T22:13:52.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.281+0000 7f53db3c2700 1 --2- 172.21.15.184:0/320720974 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f53c002ff10 0x7f53c00323d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:52.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.281+0000 7f53db3c2700 1 --2- 172.21.15.184:0/320720974 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53d4060280 0x7f53d4106780 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:52.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.281+0000 7f53db3c2700 1 -- 172.21.15.184:0/320720974 >> 172.21.15.184:0/320720974 conn(0x7f53d40f0b70 msgr2=0x7f53d40f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:52.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.281+0000 7f53db3c2700 1 -- 172.21.15.184:0/320720974 shutdown_connections 2024-02-19T22:13:52.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:52.281+0000 7f53db3c2700 1 -- 172.21.15.184:0/320720974 wait complete. 2024-02-19T22:13:52.284 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:53.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:52 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:52.280902+0000 mon.smithi176 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.184:0/320720974' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:53.877 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:53.878 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:54.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:51.877250+0000 mgr.smithi176.faoogw (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:55.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.721+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2137235773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6c0060280 msgr2=0x7fa6c00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:55.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.721+0000 7fa6c5cf8700 1 --2- 172.21.15.184:0/2137235773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6c0060280 0x7fa6c00f90f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fa6b0009fb0 tx=0x7fa6b0009b30).stop 2024-02-19T22:13:55.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.721+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2137235773 shutdown_connections 2024-02-19T22:13:55.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.721+0000 7fa6c5cf8700 1 --2- 172.21.15.184:0/2137235773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6c0060280 0x7fa6c00f90f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:55.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.721+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2137235773 >> 172.21.15.184:0/2137235773 conn(0x7fa6c00f0b70 msgr2=0x7fa6c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:55.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.721+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2137235773 shutdown_connections 2024-02-19T22:13:55.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.721+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2137235773 wait complete. 2024-02-19T22:13:55.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.721+0000 7fa6c5cf8700 1 Processor -- start 2024-02-19T22:13:55.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.722+0000 7fa6c5cf8700 1 -- start start 2024-02-19T22:13:55.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.722+0000 7fa6c5cf8700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6c0060280 0x7fa6c0106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:55.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.722+0000 7fa6c5cf8700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c0106c70 con 0x7fa6c0060280 2024-02-19T22:13:55.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.722+0000 7fa6bf7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6c0060280 0x7fa6c0106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:55.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.722+0000 7fa6bf7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6c0060280 0x7fa6c0106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:43318/0 (socket says 172.21.15.184:43318) 2024-02-19T22:13:55.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.722+0000 7fa6bf7fe700 1 -- 172.21.15.184:0/2755368807 learned_addr learned my addr 172.21.15.184:0/2755368807 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:55.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.722+0000 7fa6bf7fe700 1 -- 172.21.15.184:0/2755368807 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b0005040 con 0x7fa6c0060280 2024-02-19T22:13:55.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.722+0000 7fa6bf7fe700 1 --2- 172.21.15.184:0/2755368807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6c0060280 0x7fa6c0106780 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fa6b00089f0 tx=0x7fa6b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:55.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.722+0000 7fa6a7fff700 1 -- 172.21.15.184:0/2755368807 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6b0009320 con 0x7fa6c0060280 2024-02-19T22:13:55.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.722+0000 7fa6a7fff700 1 -- 172.21.15.184:0/2755368807 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6b0013380 con 0x7fa6c0060280 2024-02-19T22:13:55.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.722+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2755368807 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c0106e70 con 0x7fa6c0060280 2024-02-19T22:13:55.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.723+0000 7fa6a7fff700 1 -- 172.21.15.184:0/2755368807 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6b0014e20 con 0x7fa6c0060280 2024-02-19T22:13:55.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.723+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2755368807 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c0107300 con 0x7fa6c0060280 2024-02-19T22:13:55.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.723+0000 7fa6a7fff700 1 -- 172.21.15.184:0/2755368807 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6b00263a0 con 0x7fa6c0060280 2024-02-19T22:13:55.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.723+0000 7fa6a7fff700 1 --2- 172.21.15.184:0/2755368807 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa6a802ff10 0x7fa6a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:55.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.723+0000 7fa6a7fff700 1 -- 172.21.15.184:0/2755368807 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6b0041090 con 0x7fa6c0060280 2024-02-19T22:13:55.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.724+0000 7fa6beffd700 1 --2- 172.21.15.184:0/2755368807 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa6a802ff10 0x7fa6a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:55.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.724+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2755368807 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c0059100 con 0x7fa6c0060280 2024-02-19T22:13:55.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.724+0000 7fa6beffd700 1 --2- 172.21.15.184:0/2755368807 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa6a802ff10 0x7fa6a80323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fa6b40096a0 tx=0x7fa6b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:55.733 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.732+0000 7fa6a7fff700 1 -- 172.21.15.184:0/2755368807 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6b0012920 con 0x7fa6c0060280 2024-02-19T22:13:55.939 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.937+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2755368807 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6c0045ad0 con 0x7fa6c0060280 2024-02-19T22:13:55.939 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.938+0000 7fa6a7fff700 1 -- 172.21.15.184:0/2755368807 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa6b000f340 con 0x7fa6c0060280 2024-02-19T22:13:55.939 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:55.939 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:55.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.939+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2755368807 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa6a802ff10 msgr2=0x7fa6a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:55.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.939+0000 7fa6c5cf8700 1 --2- 172.21.15.184:0/2755368807 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa6a802ff10 0x7fa6a80323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fa6b40096a0 tx=0x7fa6b4006b40).stop 2024-02-19T22:13:55.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.939+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2755368807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6c0060280 msgr2=0x7fa6c0106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:55.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.939+0000 7fa6c5cf8700 1 --2- 172.21.15.184:0/2755368807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6c0060280 0x7fa6c0106780 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fa6b00089f0 tx=0x7fa6b000b040).stop 2024-02-19T22:13:55.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.939+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2755368807 shutdown_connections 2024-02-19T22:13:55.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.939+0000 7fa6c5cf8700 1 --2- 172.21.15.184:0/2755368807 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa6a802ff10 0x7fa6a80323d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:55.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.939+0000 7fa6c5cf8700 1 --2- 172.21.15.184:0/2755368807 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa6c0060280 0x7fa6c0106780 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:55.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.939+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2755368807 >> 172.21.15.184:0/2755368807 conn(0x7fa6c00f0b70 msgr2=0x7fa6c00f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:55.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.939+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2755368807 shutdown_connections 2024-02-19T22:13:55.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:55.939+0000 7fa6c5cf8700 1 -- 172.21.15.184:0/2755368807 wait complete. 2024-02-19T22:13:55.942 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:13:56.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:55 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:53.877674+0000 mgr.smithi176.faoogw (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:57.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:56 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:55.939108+0000 mon.smithi176 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.184:0/2755368807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:13:57.583 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:57.583 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:55.877928+0000 mgr.smithi176.faoogw (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:59.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.523+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/967122334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c58060220 msgr2=0x7f3c580f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:59.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.523+0000 7f3c5e9e7700 1 --2- 172.21.15.184:0/967122334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c58060220 0x7f3c580f9150 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f3c48009fb0 tx=0x7f3c48009b30).stop 2024-02-19T22:13:59.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.523+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/967122334 shutdown_connections 2024-02-19T22:13:59.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.523+0000 7f3c5e9e7700 1 --2- 172.21.15.184:0/967122334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c58060220 0x7f3c580f9150 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:59.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.523+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/967122334 >> 172.21.15.184:0/967122334 conn(0x7f3c580f0b70 msgr2=0x7f3c580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:59.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.523+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/967122334 shutdown_connections 2024-02-19T22:13:59.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.523+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/967122334 wait complete. 2024-02-19T22:13:59.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.524+0000 7f3c5e9e7700 1 Processor -- start 2024-02-19T22:13:59.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.524+0000 7f3c5e9e7700 1 -- start start 2024-02-19T22:13:59.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.524+0000 7f3c5e9e7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c58060220 0x7f3c581067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:59.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.524+0000 7f3c5e9e7700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c58106c90 con 0x7f3c58060220 2024-02-19T22:13:59.528 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.524+0000 7f3c57fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c58060220 0x7f3c581067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:59.528 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.524+0000 7f3c57fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c58060220 0x7f3c581067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:58710/0 (socket says 172.21.15.184:58710) 2024-02-19T22:13:59.528 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.524+0000 7f3c57fff700 1 -- 172.21.15.184:0/1271357522 learned_addr learned my addr 172.21.15.184:0/1271357522 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:13:59.528 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.525+0000 7f3c57fff700 1 -- 172.21.15.184:0/1271357522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c48005040 con 0x7f3c58060220 2024-02-19T22:13:59.528 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.525+0000 7f3c57fff700 1 --2- 172.21.15.184:0/1271357522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c58060220 0x7f3c581067a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3c48009830 tx=0x7f3c4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:59.528 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.525+0000 7f3c54ff9700 1 -- 172.21.15.184:0/1271357522 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c48009320 con 0x7f3c58060220 2024-02-19T22:13:59.528 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.525+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/1271357522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c58106e90 con 0x7f3c58060220 2024-02-19T22:13:59.528 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.525+0000 7f3c54ff9700 1 -- 172.21.15.184:0/1271357522 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c48013380 con 0x7f3c58060220 2024-02-19T22:13:59.528 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.525+0000 7f3c54ff9700 1 -- 172.21.15.184:0/1271357522 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c48014420 con 0x7f3c58060220 2024-02-19T22:13:59.528 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.525+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/1271357522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c58107320 con 0x7f3c58060220 2024-02-19T22:13:59.529 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.526+0000 7f3c54ff9700 1 -- 172.21.15.184:0/1271357522 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c4801e420 con 0x7f3c58060220 2024-02-19T22:13:59.529 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.526+0000 7f3c54ff9700 1 --2- 172.21.15.184:0/1271357522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3c4002ff60 0x7f3c40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:59.529 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.526+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/1271357522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c58045ad0 con 0x7f3c58060220 2024-02-19T22:13:59.529 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.526+0000 7f3c54ff9700 1 -- 172.21.15.184:0/1271357522 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c48040ef0 con 0x7f3c58060220 2024-02-19T22:13:59.529 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.526+0000 7f3c577fe700 1 --2- 172.21.15.184:0/1271357522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3c4002ff60 0x7f3c40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:59.529 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.527+0000 7f3c577fe700 1 --2- 172.21.15.184:0/1271357522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3c4002ff60 0x7f3c40032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f3c4c0096a0 tx=0x7f3c4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:59.535 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.534+0000 7f3c54ff9700 1 -- 172.21.15.184:0/1271357522 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c48004490 con 0x7f3c58060220 2024-02-19T22:13:59.738 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.736+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/1271357522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c580fc090 con 0x7f3c58060220 2024-02-19T22:13:59.738 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.737+0000 7f3c54ff9700 1 -- 172.21.15.184:0/1271357522 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3c4802c2d0 con 0x7f3c58060220 2024-02-19T22:13:59.739 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:13:59.739 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:13:59.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.738+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/1271357522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3c4002ff60 msgr2=0x7f3c40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:59.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.738+0000 7f3c5e9e7700 1 --2- 172.21.15.184:0/1271357522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3c4002ff60 0x7f3c40032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f3c4c0096a0 tx=0x7f3c4c006b40).stop 2024-02-19T22:13:59.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.738+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/1271357522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c58060220 msgr2=0x7f3c581067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:59.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.738+0000 7f3c5e9e7700 1 --2- 172.21.15.184:0/1271357522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c58060220 0x7f3c581067a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3c48009830 tx=0x7f3c4800b040).stop 2024-02-19T22:13:59.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.738+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/1271357522 shutdown_connections 2024-02-19T22:13:59.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.738+0000 7f3c5e9e7700 1 --2- 172.21.15.184:0/1271357522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3c4002ff60 0x7f3c40032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:59.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.738+0000 7f3c5e9e7700 1 --2- 172.21.15.184:0/1271357522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3c58060220 0x7f3c581067a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:59.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.738+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/1271357522 >> 172.21.15.184:0/1271357522 conn(0x7f3c580f0b70 msgr2=0x7f3c580f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:59.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.739+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/1271357522 shutdown_connections 2024-02-19T22:13:59.741 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:13:59.739+0000 7f3c5e9e7700 1 -- 172.21.15.184:0/1271357522 wait complete. 2024-02-19T22:13:59.741 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:00.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:59 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:57.878136+0000 mgr.smithi176.faoogw (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:00.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:13:59 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:13:59.738391+0000 mon.smithi176 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.184:0/1271357522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:01.409 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:01.409 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:02.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:13:59.878395+0000 mgr.smithi176.faoogw (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:03.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.231+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/3783478028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b640f5410 msgr2=0x7f2b640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:03.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.231+0000 7f2b6b7bf700 1 --2- 172.21.15.184:0/3783478028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b640f5410 0x7f2b640f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f2b60009fb0 tx=0x7f2b60009b30).stop 2024-02-19T22:14:03.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.231+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/3783478028 shutdown_connections 2024-02-19T22:14:03.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.231+0000 7f2b6b7bf700 1 --2- 172.21.15.184:0/3783478028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b640f5410 0x7f2b640f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:03.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.231+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/3783478028 >> 172.21.15.184:0/3783478028 conn(0x7f2b640f0b70 msgr2=0x7f2b640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:03.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.231+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/3783478028 shutdown_connections 2024-02-19T22:14:03.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.231+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/3783478028 wait complete. 2024-02-19T22:14:03.234 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.231+0000 7f2b6b7bf700 1 Processor -- start 2024-02-19T22:14:03.235 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.231+0000 7f2b6b7bf700 1 -- start start 2024-02-19T22:14:03.235 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.231+0000 7f2b6b7bf700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b640f5410 0x7f2b641067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:03.235 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.231+0000 7f2b6b7bf700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b64106cd0 con 0x7f2b640f5410 2024-02-19T22:14:03.235 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.232+0000 7f2b6955b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b640f5410 0x7f2b641067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:03.235 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.232+0000 7f2b6955b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b640f5410 0x7f2b641067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:58732/0 (socket says 172.21.15.184:58732) 2024-02-19T22:14:03.235 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.232+0000 7f2b6955b700 1 -- 172.21.15.184:0/2213388030 learned_addr learned my addr 172.21.15.184:0/2213388030 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:03.235 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.232+0000 7f2b6955b700 1 -- 172.21.15.184:0/2213388030 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b60005040 con 0x7f2b640f5410 2024-02-19T22:14:03.235 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.232+0000 7f2b6955b700 1 --2- 172.21.15.184:0/2213388030 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b640f5410 0x7f2b641067e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f2b60005010 tx=0x7f2b6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:03.235 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.232+0000 7f2b59ffb700 1 -- 172.21.15.184:0/2213388030 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b60009320 con 0x7f2b640f5410 2024-02-19T22:14:03.235 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.232+0000 7f2b59ffb700 1 -- 172.21.15.184:0/2213388030 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b60013380 con 0x7f2b640f5410 2024-02-19T22:14:03.235 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.232+0000 7f2b59ffb700 1 -- 172.21.15.184:0/2213388030 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b60014e50 con 0x7f2b640f5410 2024-02-19T22:14:03.235 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.232+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/2213388030 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b64106ed0 con 0x7f2b640f5410 2024-02-19T22:14:03.235 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.232+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/2213388030 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b64107360 con 0x7f2b640f5410 2024-02-19T22:14:03.236 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.233+0000 7f2b59ffb700 1 -- 172.21.15.184:0/2213388030 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b600095f0 con 0x7f2b640f5410 2024-02-19T22:14:03.236 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.233+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/2213388030 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b64045ad0 con 0x7f2b640f5410 2024-02-19T22:14:03.236 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.233+0000 7f2b59ffb700 1 --2- 172.21.15.184:0/2213388030 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2b5002ff10 0x7f2b500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:03.236 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.233+0000 7f2b59ffb700 1 -- 172.21.15.184:0/2213388030 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b60040f70 con 0x7f2b640f5410 2024-02-19T22:14:03.236 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.234+0000 7f2b68d5a700 1 --2- 172.21.15.184:0/2213388030 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2b5002ff10 0x7f2b500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:03.237 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.234+0000 7f2b68d5a700 1 --2- 172.21.15.184:0/2213388030 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2b5002ff10 0x7f2b500323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f2b540096a0 tx=0x7f2b54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:03.239 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.238+0000 7f2b59ffb700 1 -- 172.21.15.184:0/2213388030 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b6001db60 con 0x7f2b640f5410 2024-02-19T22:14:03.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.441+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/2213388030 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b640fafa0 con 0x7f2b640f5410 2024-02-19T22:14:03.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.442+0000 7f2b59ffb700 1 -- 172.21.15.184:0/2213388030 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2b6001d980 con 0x7f2b640f5410 2024-02-19T22:14:03.444 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:03.444 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:03.445 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.444+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/2213388030 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2b5002ff10 msgr2=0x7f2b500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:03.445 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.444+0000 7f2b6b7bf700 1 --2- 172.21.15.184:0/2213388030 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2b5002ff10 0x7f2b500323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f2b540096a0 tx=0x7f2b54006b40).stop 2024-02-19T22:14:03.445 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.444+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/2213388030 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b640f5410 msgr2=0x7f2b641067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:03.445 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.444+0000 7f2b6b7bf700 1 --2- 172.21.15.184:0/2213388030 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b640f5410 0x7f2b641067e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f2b60005010 tx=0x7f2b6000b040).stop 2024-02-19T22:14:03.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.444+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/2213388030 shutdown_connections 2024-02-19T22:14:03.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.444+0000 7f2b6b7bf700 1 --2- 172.21.15.184:0/2213388030 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f2b5002ff10 0x7f2b500323d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:03.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.444+0000 7f2b6b7bf700 1 --2- 172.21.15.184:0/2213388030 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2b640f5410 0x7f2b641067e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:03.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.444+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/2213388030 >> 172.21.15.184:0/2213388030 conn(0x7f2b640f0b70 msgr2=0x7f2b6405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:03.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.444+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/2213388030 shutdown_connections 2024-02-19T22:14:03.446 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:03.444+0000 7f2b6b7bf700 1 -- 172.21.15.184:0/2213388030 wait complete. 2024-02-19T22:14:03.446 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:04.078 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:01.878689+0000 mgr.smithi176.faoogw (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:04.079 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:03.443342+0000 mon.smithi176 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.184:0/2213388030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:05.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:05.090 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:05.636 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:03.878929+0000 mgr.smithi176.faoogw (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:05.636 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:04.226824+0000 mon.smithi176 (mon.0) 214 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:14:05.636 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:14:04.228204+0000 mgr.smithi176.faoogw (mgr.14162) 170 : cephadm [INF] Deploying daemon node-exporter.smithi176 on smithi176 2024-02-19T22:14:06.817 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.814+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/3372949564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2f80f62a0 msgr2=0x7fc2f80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:06.817 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.814+0000 7fc2fcb5e700 1 --2- 172.21.15.184:0/3372949564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2f80f62a0 0x7fc2f80f66c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fc2e8009fb0 tx=0x7fc2e8009b30).stop 2024-02-19T22:14:06.817 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.814+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/3372949564 shutdown_connections 2024-02-19T22:14:06.817 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.814+0000 7fc2fcb5e700 1 --2- 172.21.15.184:0/3372949564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2f80f62a0 0x7fc2f80f66c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:06.817 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.814+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/3372949564 >> 172.21.15.184:0/3372949564 conn(0x7fc2f80f1aa0 msgr2=0x7fc2f80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:06.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.814+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/3372949564 shutdown_connections 2024-02-19T22:14:06.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.814+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/3372949564 wait complete. 2024-02-19T22:14:06.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.814+0000 7fc2fcb5e700 1 Processor -- start 2024-02-19T22:14:06.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.814+0000 7fc2fcb5e700 1 -- start start 2024-02-19T22:14:06.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.815+0000 7fc2fcb5e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2f80f62a0 0x7fc2f81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:06.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.815+0000 7fc2fcb5e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f8106ac0 con 0x7fc2f80f62a0 2024-02-19T22:14:06.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.815+0000 7fc2f77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2f80f62a0 0x7fc2f81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:06.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.815+0000 7fc2f77fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2f80f62a0 0x7fc2f81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:41510/0 (socket says 172.21.15.184:41510) 2024-02-19T22:14:06.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.815+0000 7fc2f77fe700 1 -- 172.21.15.184:0/1690071522 learned_addr learned my addr 172.21.15.184:0/1690071522 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:06.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.815+0000 7fc2f77fe700 1 -- 172.21.15.184:0/1690071522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e8005040 con 0x7fc2f80f62a0 2024-02-19T22:14:06.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.815+0000 7fc2f77fe700 1 --2- 172.21.15.184:0/1690071522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2f80f62a0 0x7fc2f81065d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fc2e8009830 tx=0x7fc2e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:06.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.815+0000 7fc2dffff700 1 -- 172.21.15.184:0/1690071522 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e8009320 con 0x7fc2f80f62a0 2024-02-19T22:14:06.818 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.815+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/1690071522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f8106cc0 con 0x7fc2f80f62a0 2024-02-19T22:14:06.819 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.815+0000 7fc2dffff700 1 -- 172.21.15.184:0/1690071522 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2e8013380 con 0x7fc2f80f62a0 2024-02-19T22:14:06.819 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.815+0000 7fc2dffff700 1 -- 172.21.15.184:0/1690071522 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e8014420 con 0x7fc2f80f62a0 2024-02-19T22:14:06.819 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.815+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/1690071522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f8107150 con 0x7fc2f80f62a0 2024-02-19T22:14:06.819 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.816+0000 7fc2dffff700 1 -- 172.21.15.184:0/1690071522 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2e801e420 con 0x7fc2f80f62a0 2024-02-19T22:14:06.820 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.816+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/1690071522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f8045ad0 con 0x7fc2f80f62a0 2024-02-19T22:14:06.820 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.816+0000 7fc2dffff700 1 --2- 172.21.15.184:0/1690071522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc2e002ff60 0x7fc2e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:06.820 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.816+0000 7fc2dffff700 1 -- 172.21.15.184:0/1690071522 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2e8040e30 con 0x7fc2f80f62a0 2024-02-19T22:14:06.820 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.817+0000 7fc2f6ffd700 1 --2- 172.21.15.184:0/1690071522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc2e002ff60 0x7fc2e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:06.820 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.817+0000 7fc2f6ffd700 1 --2- 172.21.15.184:0/1690071522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc2e002ff60 0x7fc2e0032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fc2ec0096a0 tx=0x7fc2ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:06.822 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:06.821+0000 7fc2dffff700 1 -- 172.21.15.184:0/1690071522 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2e801cb10 con 0x7fc2f80f62a0 2024-02-19T22:14:07.027 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:07.026+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/1690071522 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2f80fbe30 con 0x7fc2f80f62a0 2024-02-19T22:14:07.028 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:07.026+0000 7fc2dffff700 1 -- 172.21.15.184:0/1690071522 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc2e8004490 con 0x7fc2f80f62a0 2024-02-19T22:14:07.028 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:07.028 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:07.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:07.028+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/1690071522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc2e002ff60 msgr2=0x7fc2e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:07.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:07.028+0000 7fc2fcb5e700 1 --2- 172.21.15.184:0/1690071522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc2e002ff60 0x7fc2e0032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fc2ec0096a0 tx=0x7fc2ec006b40).stop 2024-02-19T22:14:07.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:07.028+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/1690071522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2f80f62a0 msgr2=0x7fc2f81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:07.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:07.028+0000 7fc2fcb5e700 1 --2- 172.21.15.184:0/1690071522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2f80f62a0 0x7fc2f81065d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fc2e8009830 tx=0x7fc2e800b040).stop 2024-02-19T22:14:07.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:07.028+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/1690071522 shutdown_connections 2024-02-19T22:14:07.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:07.028+0000 7fc2fcb5e700 1 --2- 172.21.15.184:0/1690071522 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc2e002ff60 0x7fc2e0032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:07.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:07.028+0000 7fc2fcb5e700 1 --2- 172.21.15.184:0/1690071522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2f80f62a0 0x7fc2f81065d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:07.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:07.028+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/1690071522 >> 172.21.15.184:0/1690071522 conn(0x7fc2f80f1aa0 msgr2=0x7fc2f80f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:07.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:07.028+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/1690071522 shutdown_connections 2024-02-19T22:14:07.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:07.028+0000 7fc2fcb5e700 1 -- 172.21.15.184:0/1690071522 wait complete. 2024-02-19T22:14:07.031 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:07.635 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:05.879178+0000 mgr.smithi176.faoogw (mgr.14162) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:07.635 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:07.027864+0000 mon.smithi176 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.184:0/1690071522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:08.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:08.763 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:09.635 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:07.879501+0000 mgr.smithi176.faoogw (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:10.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.714+0000 7f9928b34700 1 -- 172.21.15.184:0/3965756006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99240f5410 msgr2=0x7f99240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:10.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.714+0000 7f9928b34700 1 --2- 172.21.15.184:0/3965756006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99240f5410 0x7f99240f5830 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f9914009fb0 tx=0x7f9914009b30).stop 2024-02-19T22:14:10.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.714+0000 7f9928b34700 1 -- 172.21.15.184:0/3965756006 shutdown_connections 2024-02-19T22:14:10.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.714+0000 7f9928b34700 1 --2- 172.21.15.184:0/3965756006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99240f5410 0x7f99240f5830 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:10.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.714+0000 7f9928b34700 1 -- 172.21.15.184:0/3965756006 >> 172.21.15.184:0/3965756006 conn(0x7f99240f0b70 msgr2=0x7f99240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:10.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.714+0000 7f9928b34700 1 -- 172.21.15.184:0/3965756006 shutdown_connections 2024-02-19T22:14:10.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.714+0000 7f9928b34700 1 -- 172.21.15.184:0/3965756006 wait complete. 2024-02-19T22:14:10.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.715+0000 7f9928b34700 1 Processor -- start 2024-02-19T22:14:10.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.715+0000 7f9928b34700 1 -- start start 2024-02-19T22:14:10.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.715+0000 7f9928b34700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99240f5410 0x7f99241067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:10.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.715+0000 7f9928b34700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9924106ca0 con 0x7f99240f5410 2024-02-19T22:14:10.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.715+0000 7f992259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99240f5410 0x7f99241067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:10.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.715+0000 7f992259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99240f5410 0x7f99241067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:41518/0 (socket says 172.21.15.184:41518) 2024-02-19T22:14:10.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.715+0000 7f992259c700 1 -- 172.21.15.184:0/3732386603 learned_addr learned my addr 172.21.15.184:0/3732386603 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:10.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.716+0000 7f992259c700 1 -- 172.21.15.184:0/3732386603 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9914005040 con 0x7f99240f5410 2024-02-19T22:14:10.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.716+0000 7f992259c700 1 --2- 172.21.15.184:0/3732386603 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99240f5410 0x7f99241067b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f9914005010 tx=0x7f991400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:10.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.716+0000 7f9912ffd700 1 -- 172.21.15.184:0/3732386603 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9914009320 con 0x7f99240f5410 2024-02-19T22:14:10.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.716+0000 7f9928b34700 1 -- 172.21.15.184:0/3732386603 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9924106ea0 con 0x7f99240f5410 2024-02-19T22:14:10.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.716+0000 7f9912ffd700 1 -- 172.21.15.184:0/3732386603 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9914013380 con 0x7f99240f5410 2024-02-19T22:14:10.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.716+0000 7f9912ffd700 1 -- 172.21.15.184:0/3732386603 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9914014e50 con 0x7f99240f5410 2024-02-19T22:14:10.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.716+0000 7f9928b34700 1 -- 172.21.15.184:0/3732386603 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9924107330 con 0x7f99240f5410 2024-02-19T22:14:10.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.717+0000 7f9928b34700 1 -- 172.21.15.184:0/3732386603 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9924045ad0 con 0x7f99240f5410 2024-02-19T22:14:10.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.717+0000 7f9912ffd700 1 -- 172.21.15.184:0/3732386603 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99140095f0 con 0x7f99240f5410 2024-02-19T22:14:10.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.717+0000 7f9912ffd700 1 --2- 172.21.15.184:0/3732386603 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f990c02ff10 0x7f990c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:10.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.717+0000 7f9912ffd700 1 -- 172.21.15.184:0/3732386603 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9914040e60 con 0x7f99240f5410 2024-02-19T22:14:10.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.717+0000 7f9921d9b700 1 --2- 172.21.15.184:0/3732386603 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f990c02ff10 0x7f990c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:10.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.718+0000 7f9921d9b700 1 --2- 172.21.15.184:0/3732386603 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f990c02ff10 0x7f990c0323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f99180096a0 tx=0x7f9918006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:10.723 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.722+0000 7f9912ffd700 1 -- 172.21.15.184:0/3732386603 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9914014420 con 0x7f99240f5410 2024-02-19T22:14:10.928 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.927+0000 7f9928b34700 1 -- 172.21.15.184:0/3732386603 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99240fafa0 con 0x7f99240f5410 2024-02-19T22:14:10.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.928+0000 7f9912ffd700 1 -- 172.21.15.184:0/3732386603 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9914010230 con 0x7f99240f5410 2024-02-19T22:14:10.929 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:10.929 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:10.931 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.929+0000 7f9928b34700 1 -- 172.21.15.184:0/3732386603 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f990c02ff10 msgr2=0x7f990c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:10.931 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.929+0000 7f9928b34700 1 --2- 172.21.15.184:0/3732386603 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f990c02ff10 0x7f990c0323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f99180096a0 tx=0x7f9918006b40).stop 2024-02-19T22:14:10.931 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.929+0000 7f9928b34700 1 -- 172.21.15.184:0/3732386603 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99240f5410 msgr2=0x7f99241067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:10.931 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.929+0000 7f9928b34700 1 --2- 172.21.15.184:0/3732386603 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99240f5410 0x7f99241067b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f9914005010 tx=0x7f991400b040).stop 2024-02-19T22:14:10.931 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.929+0000 7f9928b34700 1 -- 172.21.15.184:0/3732386603 shutdown_connections 2024-02-19T22:14:10.931 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.929+0000 7f9928b34700 1 --2- 172.21.15.184:0/3732386603 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f990c02ff10 0x7f990c0323d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:10.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.929+0000 7f9928b34700 1 --2- 172.21.15.184:0/3732386603 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f99240f5410 0x7f99241067b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:10.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.929+0000 7f9928b34700 1 -- 172.21.15.184:0/3732386603 >> 172.21.15.184:0/3732386603 conn(0x7f99240f0b70 msgr2=0x7f992405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:10.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.929+0000 7f9928b34700 1 -- 172.21.15.184:0/3732386603 shutdown_connections 2024-02-19T22:14:10.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:10.929+0000 7f9928b34700 1 -- 172.21.15.184:0/3732386603 wait complete. 2024-02-19T22:14:10.932 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:12.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:09.879830+0000 mgr.smithi176.faoogw (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:12.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:10.929155+0000 mon.smithi176 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.184:0/3732386603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:12.631 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:12.631 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:14.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:13 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:11.880121+0000 mgr.smithi176.faoogw (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:14.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.586+0000 7f39e299b700 1 -- 172.21.15.184:0/1855854087 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39dc0f5410 msgr2=0x7f39dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:14.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.586+0000 7f39e299b700 1 --2- 172.21.15.184:0/1855854087 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39dc0f5410 0x7f39dc0f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f39cc009fb0 tx=0x7f39cc009b30).stop 2024-02-19T22:14:14.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.586+0000 7f39e299b700 1 -- 172.21.15.184:0/1855854087 shutdown_connections 2024-02-19T22:14:14.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.586+0000 7f39e299b700 1 --2- 172.21.15.184:0/1855854087 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39dc0f5410 0x7f39dc0f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:14.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.586+0000 7f39e299b700 1 -- 172.21.15.184:0/1855854087 >> 172.21.15.184:0/1855854087 conn(0x7f39dc0f0b70 msgr2=0x7f39dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:14.595 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.586+0000 7f39e299b700 1 -- 172.21.15.184:0/1855854087 shutdown_connections 2024-02-19T22:14:14.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.586+0000 7f39e299b700 1 -- 172.21.15.184:0/1855854087 wait complete. 2024-02-19T22:14:14.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.586+0000 7f39e299b700 1 Processor -- start 2024-02-19T22:14:14.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.587+0000 7f39e299b700 1 -- start start 2024-02-19T22:14:14.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.587+0000 7f39e299b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39dc0f5410 0x7f39dc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:14.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.587+0000 7f39e299b700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc105bd0 con 0x7f39dc0f5410 2024-02-19T22:14:14.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.587+0000 7f39dbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39dc0f5410 0x7f39dc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:14.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.587+0000 7f39dbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39dc0f5410 0x7f39dc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:41524/0 (socket says 172.21.15.184:41524) 2024-02-19T22:14:14.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.587+0000 7f39dbfff700 1 -- 172.21.15.184:0/2239708798 learned_addr learned my addr 172.21.15.184:0/2239708798 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:14.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.587+0000 7f39dbfff700 1 -- 172.21.15.184:0/2239708798 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc005040 con 0x7f39dc0f5410 2024-02-19T22:14:14.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.587+0000 7f39dbfff700 1 --2- 172.21.15.184:0/2239708798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39dc0f5410 0x7f39dc1056e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f39cc005010 tx=0x7f39cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:14.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.587+0000 7f39d8ff9700 1 -- 172.21.15.184:0/2239708798 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39cc009320 con 0x7f39dc0f5410 2024-02-19T22:14:14.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.587+0000 7f39e299b700 1 -- 172.21.15.184:0/2239708798 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39dc105dd0 con 0x7f39dc0f5410 2024-02-19T22:14:14.597 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.587+0000 7f39d8ff9700 1 -- 172.21.15.184:0/2239708798 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39cc013380 con 0x7f39dc0f5410 2024-02-19T22:14:14.597 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.587+0000 7f39d8ff9700 1 -- 172.21.15.184:0/2239708798 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39cc014e50 con 0x7f39dc0f5410 2024-02-19T22:14:14.597 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.587+0000 7f39e299b700 1 -- 172.21.15.184:0/2239708798 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39dc106260 con 0x7f39dc0f5410 2024-02-19T22:14:14.597 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.588+0000 7f39d8ff9700 1 -- 172.21.15.184:0/2239708798 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39cc0095f0 con 0x7f39dc0f5410 2024-02-19T22:14:14.597 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.588+0000 7f39e299b700 1 -- 172.21.15.184:0/2239708798 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39dc045ad0 con 0x7f39dc0f5410 2024-02-19T22:14:14.597 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.589+0000 7f39d8ff9700 1 --2- 172.21.15.184:0/2239708798 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f39c402ff10 0x7f39c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:14.597 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.589+0000 7f39d8ff9700 1 -- 172.21.15.184:0/2239708798 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39cc040e90 con 0x7f39dc0f5410 2024-02-19T22:14:14.597 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.589+0000 7f39db7fe700 1 --2- 172.21.15.184:0/2239708798 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f39c402ff10 0x7f39c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:14.597 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.589+0000 7f39db7fe700 1 --2- 172.21.15.184:0/2239708798 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f39c402ff10 0x7f39c40323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f39d00096a0 tx=0x7f39d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:14.597 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.593+0000 7f39d8ff9700 1 -- 172.21.15.184:0/2239708798 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39cc014420 con 0x7f39dc0f5410 2024-02-19T22:14:14.798 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.796+0000 7f39e299b700 1 -- 172.21.15.184:0/2239708798 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39dc107110 con 0x7f39dc0f5410 2024-02-19T22:14:14.798 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.797+0000 7f39d8ff9700 1 -- 172.21.15.184:0/2239708798 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f39cc010230 con 0x7f39dc0f5410 2024-02-19T22:14:14.798 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:14.799 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:14.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.798+0000 7f39e299b700 1 -- 172.21.15.184:0/2239708798 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f39c402ff10 msgr2=0x7f39c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:14.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.798+0000 7f39e299b700 1 --2- 172.21.15.184:0/2239708798 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f39c402ff10 0x7f39c40323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f39d00096a0 tx=0x7f39d0006b40).stop 2024-02-19T22:14:14.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.798+0000 7f39e299b700 1 -- 172.21.15.184:0/2239708798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39dc0f5410 msgr2=0x7f39dc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:14.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.798+0000 7f39e299b700 1 --2- 172.21.15.184:0/2239708798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39dc0f5410 0x7f39dc1056e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f39cc005010 tx=0x7f39cc00b040).stop 2024-02-19T22:14:14.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.798+0000 7f39e299b700 1 -- 172.21.15.184:0/2239708798 shutdown_connections 2024-02-19T22:14:14.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.798+0000 7f39e299b700 1 --2- 172.21.15.184:0/2239708798 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f39c402ff10 0x7f39c40323d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:14.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.798+0000 7f39e299b700 1 --2- 172.21.15.184:0/2239708798 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39dc0f5410 0x7f39dc1056e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:14.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.798+0000 7f39e299b700 1 -- 172.21.15.184:0/2239708798 >> 172.21.15.184:0/2239708798 conn(0x7f39dc0f0b70 msgr2=0x7f39dc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:14.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.798+0000 7f39e299b700 1 -- 172.21.15.184:0/2239708798 shutdown_connections 2024-02-19T22:14:14.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:14.798+0000 7f39e299b700 1 -- 172.21.15.184:0/2239708798 wait complete. 2024-02-19T22:14:14.801 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:16.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:13.880464+0000 mgr.smithi176.faoogw (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:16.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:14.798195+0000 mon.smithi176 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.184:0/2239708798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:16.473 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:16.474 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:18.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:17 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:15.880797+0000 mgr.smithi176.faoogw (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:18.418 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.415+0000 7f7d09782700 1 -- 172.21.15.184:0/2080594840 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d040f5410 msgr2=0x7f7d040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:18.418 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.415+0000 7f7d09782700 1 --2- 172.21.15.184:0/2080594840 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d040f5410 0x7f7d040f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f7cf4009fb0 tx=0x7f7cf4009b30).stop 2024-02-19T22:14:18.418 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.415+0000 7f7d09782700 1 -- 172.21.15.184:0/2080594840 shutdown_connections 2024-02-19T22:14:18.418 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.415+0000 7f7d09782700 1 --2- 172.21.15.184:0/2080594840 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d040f5410 0x7f7d040f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:18.418 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.415+0000 7f7d09782700 1 -- 172.21.15.184:0/2080594840 >> 172.21.15.184:0/2080594840 conn(0x7f7d040f0b70 msgr2=0x7f7d040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:18.418 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.415+0000 7f7d09782700 1 -- 172.21.15.184:0/2080594840 shutdown_connections 2024-02-19T22:14:18.418 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.415+0000 7f7d09782700 1 -- 172.21.15.184:0/2080594840 wait complete. 2024-02-19T22:14:18.418 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.416+0000 7f7d09782700 1 Processor -- start 2024-02-19T22:14:18.418 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.416+0000 7f7d09782700 1 -- start start 2024-02-19T22:14:18.418 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.416+0000 7f7d09782700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d040f5410 0x7f7d041067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:18.419 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.416+0000 7f7d09782700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d04106ca0 con 0x7f7d040f5410 2024-02-19T22:14:18.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.417+0000 7f7d02ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d040f5410 0x7f7d041067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:18.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.417+0000 7f7d02ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d040f5410 0x7f7d041067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:53408/0 (socket says 172.21.15.184:53408) 2024-02-19T22:14:18.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.417+0000 7f7d02ffd700 1 -- 172.21.15.184:0/1984946977 learned_addr learned my addr 172.21.15.184:0/1984946977 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:18.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.417+0000 7f7d02ffd700 1 -- 172.21.15.184:0/1984946977 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf4005040 con 0x7f7d040f5410 2024-02-19T22:14:18.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.417+0000 7f7d02ffd700 1 --2- 172.21.15.184:0/1984946977 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d040f5410 0x7f7d041067b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f7cf4005010 tx=0x7f7cf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:18.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.417+0000 7f7ceb7fe700 1 -- 172.21.15.184:0/1984946977 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cf4009320 con 0x7f7d040f5410 2024-02-19T22:14:18.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.417+0000 7f7d09782700 1 -- 172.21.15.184:0/1984946977 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d04106ea0 con 0x7f7d040f5410 2024-02-19T22:14:18.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.417+0000 7f7ceb7fe700 1 -- 172.21.15.184:0/1984946977 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7cf4013380 con 0x7f7d040f5410 2024-02-19T22:14:18.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.417+0000 7f7ceb7fe700 1 -- 172.21.15.184:0/1984946977 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cf4014e50 con 0x7f7d040f5410 2024-02-19T22:14:18.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.417+0000 7f7d09782700 1 -- 172.21.15.184:0/1984946977 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d04107330 con 0x7f7d040f5410 2024-02-19T22:14:18.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.418+0000 7f7ceb7fe700 1 -- 172.21.15.184:0/1984946977 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7cf40095f0 con 0x7f7d040f5410 2024-02-19T22:14:18.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.418+0000 7f7d09782700 1 -- 172.21.15.184:0/1984946977 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d04045ad0 con 0x7f7d040f5410 2024-02-19T22:14:18.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.418+0000 7f7ceb7fe700 1 --2- 172.21.15.184:0/1984946977 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7cec02ff10 0x7f7cec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:18.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.418+0000 7f7ceb7fe700 1 -- 172.21.15.184:0/1984946977 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7cf4040e70 con 0x7f7d040f5410 2024-02-19T22:14:18.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.418+0000 7f7d027fc700 1 --2- 172.21.15.184:0/1984946977 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7cec02ff10 0x7f7cec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:18.422 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.419+0000 7f7d027fc700 1 --2- 172.21.15.184:0/1984946977 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7cec02ff10 0x7f7cec0323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f7cf80096a0 tx=0x7f7cf8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:18.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.423+0000 7f7ceb7fe700 1 -- 172.21.15.184:0/1984946977 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7cf4014420 con 0x7f7d040f5410 2024-02-19T22:14:18.630 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.628+0000 7f7d09782700 1 -- 172.21.15.184:0/1984946977 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d040fafa0 con 0x7f7d040f5410 2024-02-19T22:14:18.630 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.629+0000 7f7ceb7fe700 1 -- 172.21.15.184:0/1984946977 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7cf4010230 con 0x7f7d040f5410 2024-02-19T22:14:18.631 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:18.631 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:18.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.631+0000 7f7d09782700 1 -- 172.21.15.184:0/1984946977 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7cec02ff10 msgr2=0x7f7cec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:18.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.631+0000 7f7d09782700 1 --2- 172.21.15.184:0/1984946977 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7cec02ff10 0x7f7cec0323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f7cf80096a0 tx=0x7f7cf8006b40).stop 2024-02-19T22:14:18.632 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.631+0000 7f7d09782700 1 -- 172.21.15.184:0/1984946977 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d040f5410 msgr2=0x7f7d041067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:18.633 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.631+0000 7f7d09782700 1 --2- 172.21.15.184:0/1984946977 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d040f5410 0x7f7d041067b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f7cf4005010 tx=0x7f7cf400b040).stop 2024-02-19T22:14:18.633 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.631+0000 7f7d09782700 1 -- 172.21.15.184:0/1984946977 shutdown_connections 2024-02-19T22:14:18.633 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.631+0000 7f7d09782700 1 --2- 172.21.15.184:0/1984946977 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f7cec02ff10 0x7f7cec0323d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:18.633 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.631+0000 7f7d09782700 1 --2- 172.21.15.184:0/1984946977 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7d040f5410 0x7f7d041067b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:18.633 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.631+0000 7f7d09782700 1 -- 172.21.15.184:0/1984946977 >> 172.21.15.184:0/1984946977 conn(0x7f7d040f0b70 msgr2=0x7f7d0405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:18.633 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.631+0000 7f7d09782700 1 -- 172.21.15.184:0/1984946977 shutdown_connections 2024-02-19T22:14:18.633 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:18.631+0000 7f7d09782700 1 -- 172.21.15.184:0/1984946977 wait complete. 2024-02-19T22:14:18.633 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:19.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:18 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:18.630343+0000 mon.smithi176 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.184:0/1984946977' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:20.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:17.881141+0000 mgr.smithi176.faoogw (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:20.509 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:20.509 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:22.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:19.881432+0000 mgr.smithi176.faoogw (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:22.363 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.361+0000 7fd7b1383700 1 -- 172.21.15.184:0/128328108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7ac0f4740 msgr2=0x7fd7ac0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:22.363 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.361+0000 7fd7b1383700 1 --2- 172.21.15.184:0/128328108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7ac0f4740 0x7fd7ac0f4b60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fd7a0009fb0 tx=0x7fd7a0009b30).stop 2024-02-19T22:14:22.363 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.361+0000 7fd7b1383700 1 -- 172.21.15.184:0/128328108 shutdown_connections 2024-02-19T22:14:22.363 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.361+0000 7fd7b1383700 1 --2- 172.21.15.184:0/128328108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7ac0f4740 0x7fd7ac0f4b60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:22.363 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.361+0000 7fd7b1383700 1 -- 172.21.15.184:0/128328108 >> 172.21.15.184:0/128328108 conn(0x7fd7ac0f0260 msgr2=0x7fd7ac0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:22.363 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.361+0000 7fd7b1383700 1 -- 172.21.15.184:0/128328108 shutdown_connections 2024-02-19T22:14:22.363 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.361+0000 7fd7b1383700 1 -- 172.21.15.184:0/128328108 wait complete. 2024-02-19T22:14:22.365 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.362+0000 7fd7b1383700 1 Processor -- start 2024-02-19T22:14:22.365 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.362+0000 7fd7b1383700 1 -- start start 2024-02-19T22:14:22.365 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.362+0000 7fd7b1383700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7ac1068e0 0x7fd7ac106d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:22.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.362+0000 7fd7b1383700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac1071f0 con 0x7fd7ac1068e0 2024-02-19T22:14:22.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.362+0000 7fd7aaffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7ac1068e0 0x7fd7ac106d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:22.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.362+0000 7fd7aaffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7ac1068e0 0x7fd7ac106d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:53426/0 (socket says 172.21.15.184:53426) 2024-02-19T22:14:22.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.362+0000 7fd7aaffd700 1 -- 172.21.15.184:0/3894557191 learned_addr learned my addr 172.21.15.184:0/3894557191 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:22.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.362+0000 7fd7aaffd700 1 -- 172.21.15.184:0/3894557191 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a0005040 con 0x7fd7ac1068e0 2024-02-19T22:14:22.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.363+0000 7fd7aaffd700 1 --2- 172.21.15.184:0/3894557191 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7ac1068e0 0x7fd7ac106d00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fd7ac05ead0 tx=0x7fd7a0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:22.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.363+0000 7fd7937fe700 1 -- 172.21.15.184:0/3894557191 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7a0013380 con 0x7fd7ac1068e0 2024-02-19T22:14:22.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.363+0000 7fd7b1383700 1 -- 172.21.15.184:0/3894557191 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ac1073f0 con 0x7fd7ac1068e0 2024-02-19T22:14:22.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.363+0000 7fd7937fe700 1 -- 172.21.15.184:0/3894557191 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd7a0013790 con 0x7fd7ac1068e0 2024-02-19T22:14:22.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.363+0000 7fd7937fe700 1 -- 172.21.15.184:0/3894557191 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7a001ee70 con 0x7fd7ac1068e0 2024-02-19T22:14:22.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.363+0000 7fd7b1383700 1 -- 172.21.15.184:0/3894557191 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ac107f40 con 0x7fd7ac1068e0 2024-02-19T22:14:22.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.363+0000 7fd7937fe700 1 -- 172.21.15.184:0/3894557191 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7a0025060 con 0x7fd7ac1068e0 2024-02-19T22:14:22.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.364+0000 7fd7937fe700 1 --2- 172.21.15.184:0/3894557191 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd79402ff60 0x7fd794032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:22.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.364+0000 7fd7937fe700 1 -- 172.21.15.184:0/3894557191 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd7a0040a40 con 0x7fd7ac1068e0 2024-02-19T22:14:22.367 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.364+0000 7fd7b1383700 1 -- 172.21.15.184:0/3894557191 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7ac059100 con 0x7fd7ac1068e0 2024-02-19T22:14:22.367 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.364+0000 7fd7aa7fc700 1 --2- 172.21.15.184:0/3894557191 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd79402ff60 0x7fd794032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:22.367 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.364+0000 7fd7aa7fc700 1 --2- 172.21.15.184:0/3894557191 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd79402ff60 0x7fd794032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fd79c0096a0 tx=0x7fd79c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:22.370 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.369+0000 7fd7937fe700 1 -- 172.21.15.184:0/3894557191 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7a0024260 con 0x7fd7ac1068e0 2024-02-19T22:14:22.576 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.574+0000 7fd7b1383700 1 -- 172.21.15.184:0/3894557191 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7ac1085a0 con 0x7fd7ac1068e0 2024-02-19T22:14:22.576 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.575+0000 7fd7937fe700 1 -- 172.21.15.184:0/3894557191 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd7a0024260 con 0x7fd7ac1068e0 2024-02-19T22:14:22.577 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:22.577 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:22.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.576+0000 7fd7b1383700 1 -- 172.21.15.184:0/3894557191 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd79402ff60 msgr2=0x7fd794032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:22.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.576+0000 7fd7b1383700 1 --2- 172.21.15.184:0/3894557191 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd79402ff60 0x7fd794032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fd79c0096a0 tx=0x7fd79c006b40).stop 2024-02-19T22:14:22.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.576+0000 7fd7b1383700 1 -- 172.21.15.184:0/3894557191 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7ac1068e0 msgr2=0x7fd7ac106d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:22.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.576+0000 7fd7b1383700 1 --2- 172.21.15.184:0/3894557191 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7ac1068e0 0x7fd7ac106d00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fd7ac05ead0 tx=0x7fd7a0009b30).stop 2024-02-19T22:14:22.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.576+0000 7fd7b1383700 1 -- 172.21.15.184:0/3894557191 shutdown_connections 2024-02-19T22:14:22.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.576+0000 7fd7b1383700 1 --2- 172.21.15.184:0/3894557191 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd79402ff60 0x7fd794032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:22.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.576+0000 7fd7b1383700 1 --2- 172.21.15.184:0/3894557191 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd7ac1068e0 0x7fd7ac106d00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:22.579 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.576+0000 7fd7b1383700 1 -- 172.21.15.184:0/3894557191 >> 172.21.15.184:0/3894557191 conn(0x7fd7ac0f0260 msgr2=0x7fd7ac0f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:22.579 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.576+0000 7fd7b1383700 1 -- 172.21.15.184:0/3894557191 shutdown_connections 2024-02-19T22:14:22.579 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:22.576+0000 7fd7b1383700 1 -- 172.21.15.184:0/3894557191 wait complete. 2024-02-19T22:14:22.579 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:23.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:22 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:22.576194+0000 mon.smithi176 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.184:0/3894557191' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:24.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:21.881759+0000 mgr.smithi176.faoogw (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:24.276 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:24.276 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:26.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:23.882136+0000 mgr.smithi176.faoogw (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:26.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.158+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3979930737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1b00f6510 msgr2=0x7fe1b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:26.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.158+0000 7fe1b7ffc700 1 --2- 172.21.15.184:0/3979930737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1b00f6510 0x7fe1b00f6930 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fe1a4009fb0 tx=0x7fe1a4009b30).stop 2024-02-19T22:14:26.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.158+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3979930737 shutdown_connections 2024-02-19T22:14:26.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.158+0000 7fe1b7ffc700 1 --2- 172.21.15.184:0/3979930737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1b00f6510 0x7fe1b00f6930 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:26.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.158+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3979930737 >> 172.21.15.184:0/3979930737 conn(0x7fe1b00f1c70 msgr2=0x7fe1b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:26.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.158+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3979930737 shutdown_connections 2024-02-19T22:14:26.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.158+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3979930737 wait complete. 2024-02-19T22:14:26.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.159+0000 7fe1b7ffc700 1 Processor -- start 2024-02-19T22:14:26.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.159+0000 7fe1b7ffc700 1 -- start start 2024-02-19T22:14:26.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.159+0000 7fe1b7ffc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1b00f6510 0x7fe1b01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:26.160 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.159+0000 7fe1b7ffc700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b0106cd0 con 0x7fe1b00f6510 2024-02-19T22:14:26.161 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.159+0000 7fe1b5d98700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1b00f6510 0x7fe1b01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:26.161 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.159+0000 7fe1b5d98700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1b00f6510 0x7fe1b01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:53430/0 (socket says 172.21.15.184:53430) 2024-02-19T22:14:26.161 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.159+0000 7fe1b5d98700 1 -- 172.21.15.184:0/3340547817 learned_addr learned my addr 172.21.15.184:0/3340547817 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:26.162 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.160+0000 7fe1b5d98700 1 -- 172.21.15.184:0/3340547817 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a4005040 con 0x7fe1b00f6510 2024-02-19T22:14:26.162 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.160+0000 7fe1b5d98700 1 --2- 172.21.15.184:0/3340547817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1b00f6510 0x7fe1b01067e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fe1a4005010 tx=0x7fe1a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:26.162 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.160+0000 7fe1a27fc700 1 -- 172.21.15.184:0/3340547817 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1a4009320 con 0x7fe1b00f6510 2024-02-19T22:14:26.162 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.160+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3340547817 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b0106ed0 con 0x7fe1b00f6510 2024-02-19T22:14:26.162 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.160+0000 7fe1a27fc700 1 -- 172.21.15.184:0/3340547817 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe1a4013380 con 0x7fe1b00f6510 2024-02-19T22:14:26.162 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.160+0000 7fe1a27fc700 1 -- 172.21.15.184:0/3340547817 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1a4014e50 con 0x7fe1b00f6510 2024-02-19T22:14:26.162 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.160+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3340547817 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b0107360 con 0x7fe1b00f6510 2024-02-19T22:14:26.162 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.161+0000 7fe1a27fc700 1 -- 172.21.15.184:0/3340547817 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1a40095f0 con 0x7fe1b00f6510 2024-02-19T22:14:26.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.161+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3340547817 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b0045ad0 con 0x7fe1b00f6510 2024-02-19T22:14:26.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.161+0000 7fe1a27fc700 1 --2- 172.21.15.184:0/3340547817 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe19c02ff10 0x7fe19c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:26.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.161+0000 7fe1a27fc700 1 -- 172.21.15.184:0/3340547817 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe1a4040e70 con 0x7fe1b00f6510 2024-02-19T22:14:26.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.161+0000 7fe1b5597700 1 --2- 172.21.15.184:0/3340547817 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe19c02ff10 0x7fe19c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:26.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.162+0000 7fe1b5597700 1 --2- 172.21.15.184:0/3340547817 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe19c02ff10 0x7fe19c0323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fe1ac0096a0 tx=0x7fe1ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:26.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.165+0000 7fe1a27fc700 1 -- 172.21.15.184:0/3340547817 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1a4014420 con 0x7fe1b00f6510 2024-02-19T22:14:26.371 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.370+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3340547817 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1b0108210 con 0x7fe1b00f6510 2024-02-19T22:14:26.372 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.371+0000 7fe1a27fc700 1 -- 172.21.15.184:0/3340547817 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe1a4010230 con 0x7fe1b00f6510 2024-02-19T22:14:26.372 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:26.372 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:26.374 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.373+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3340547817 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe19c02ff10 msgr2=0x7fe19c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:26.374 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.373+0000 7fe1b7ffc700 1 --2- 172.21.15.184:0/3340547817 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe19c02ff10 0x7fe19c0323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fe1ac0096a0 tx=0x7fe1ac006b40).stop 2024-02-19T22:14:26.374 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.373+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3340547817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1b00f6510 msgr2=0x7fe1b01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:26.375 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.373+0000 7fe1b7ffc700 1 --2- 172.21.15.184:0/3340547817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1b00f6510 0x7fe1b01067e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fe1a4005010 tx=0x7fe1a400b040).stop 2024-02-19T22:14:26.375 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.373+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3340547817 shutdown_connections 2024-02-19T22:14:26.375 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.373+0000 7fe1b7ffc700 1 --2- 172.21.15.184:0/3340547817 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe19c02ff10 0x7fe19c0323d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:26.375 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.373+0000 7fe1b7ffc700 1 --2- 172.21.15.184:0/3340547817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe1b00f6510 0x7fe1b01067e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:26.375 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.373+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3340547817 >> 172.21.15.184:0/3340547817 conn(0x7fe1b00f1c70 msgr2=0x7fe1b00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:26.375 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.373+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3340547817 shutdown_connections 2024-02-19T22:14:26.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:26.373+0000 7fe1b7ffc700 1 -- 172.21.15.184:0/3340547817 wait complete. 2024-02-19T22:14:26.376 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:27.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:26 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:26.371400+0000 mon.smithi176 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.184:0/3340547817' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:28.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:25.882499+0000 mgr.smithi176.faoogw (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:28.137 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:28.138 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:30.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:29 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:27.882879+0000 mgr.smithi176.faoogw (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:30.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.060+0000 7f3b74bae700 1 -- 172.21.15.184:0/3185024656 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b700f6510 msgr2=0x7f3b700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:30.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.060+0000 7f3b74bae700 1 --2- 172.21.15.184:0/3185024656 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b700f6510 0x7f3b700f6930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f3b60009fb0 tx=0x7f3b60009b30).stop 2024-02-19T22:14:30.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.060+0000 7f3b74bae700 1 -- 172.21.15.184:0/3185024656 shutdown_connections 2024-02-19T22:14:30.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.060+0000 7f3b74bae700 1 --2- 172.21.15.184:0/3185024656 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b700f6510 0x7f3b700f6930 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:30.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.060+0000 7f3b74bae700 1 -- 172.21.15.184:0/3185024656 >> 172.21.15.184:0/3185024656 conn(0x7f3b700f1c70 msgr2=0x7f3b700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:30.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.060+0000 7f3b74bae700 1 -- 172.21.15.184:0/3185024656 shutdown_connections 2024-02-19T22:14:30.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.060+0000 7f3b74bae700 1 -- 172.21.15.184:0/3185024656 wait complete. 2024-02-19T22:14:30.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.060+0000 7f3b74bae700 1 Processor -- start 2024-02-19T22:14:30.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.060+0000 7f3b74bae700 1 -- start start 2024-02-19T22:14:30.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.061+0000 7f3b74bae700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b700f6510 0x7f3b70108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:30.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.061+0000 7f3b74bae700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b70108ef0 con 0x7f3b700f6510 2024-02-19T22:14:30.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.061+0000 7f3b6e59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b700f6510 0x7f3b70108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:30.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.061+0000 7f3b6e59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b700f6510 0x7f3b70108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:36138/0 (socket says 172.21.15.184:36138) 2024-02-19T22:14:30.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.061+0000 7f3b6e59c700 1 -- 172.21.15.184:0/1900081131 learned_addr learned my addr 172.21.15.184:0/1900081131 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:30.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.061+0000 7f3b6e59c700 1 -- 172.21.15.184:0/1900081131 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b60005040 con 0x7f3b700f6510 2024-02-19T22:14:30.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.061+0000 7f3b6e59c700 1 --2- 172.21.15.184:0/1900081131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b700f6510 0x7f3b70108a00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f3b60005010 tx=0x7f3b6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:30.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.061+0000 7f3b5effd700 1 -- 172.21.15.184:0/1900081131 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b60009320 con 0x7f3b700f6510 2024-02-19T22:14:30.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.061+0000 7f3b5effd700 1 -- 172.21.15.184:0/1900081131 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b60013380 con 0x7f3b700f6510 2024-02-19T22:14:30.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.061+0000 7f3b74bae700 1 -- 172.21.15.184:0/1900081131 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b701090f0 con 0x7f3b700f6510 2024-02-19T22:14:30.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.061+0000 7f3b5effd700 1 -- 172.21.15.184:0/1900081131 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b60014e50 con 0x7f3b700f6510 2024-02-19T22:14:30.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.062+0000 7f3b74bae700 1 -- 172.21.15.184:0/1900081131 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b70109580 con 0x7f3b700f6510 2024-02-19T22:14:30.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.062+0000 7f3b5effd700 1 -- 172.21.15.184:0/1900081131 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b600095f0 con 0x7f3b700f6510 2024-02-19T22:14:30.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.062+0000 7f3b74bae700 1 -- 172.21.15.184:0/1900081131 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b70045ad0 con 0x7f3b700f6510 2024-02-19T22:14:30.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.062+0000 7f3b5effd700 1 --2- 172.21.15.184:0/1900081131 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3b5802ff10 0x7f3b580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:30.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.062+0000 7f3b5effd700 1 -- 172.21.15.184:0/1900081131 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b60040fa0 con 0x7f3b700f6510 2024-02-19T22:14:30.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.063+0000 7f3b6dd9b700 1 --2- 172.21.15.184:0/1900081131 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3b5802ff10 0x7f3b580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:30.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.063+0000 7f3b6dd9b700 1 --2- 172.21.15.184:0/1900081131 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3b5802ff10 0x7f3b580323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3b640096a0 tx=0x7f3b64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:30.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.067+0000 7f3b5effd700 1 -- 172.21.15.184:0/1900081131 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b60014420 con 0x7f3b700f6510 2024-02-19T22:14:30.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.272+0000 7f3b74bae700 1 -- 172.21.15.184:0/1900081131 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b7010a430 con 0x7f3b700f6510 2024-02-19T22:14:30.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.273+0000 7f3b5effd700 1 -- 172.21.15.184:0/1900081131 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3b60010230 con 0x7f3b700f6510 2024-02-19T22:14:30.275 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:30.275 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:30.276 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.275+0000 7f3b74bae700 1 -- 172.21.15.184:0/1900081131 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3b5802ff10 msgr2=0x7f3b580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:30.276 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.275+0000 7f3b74bae700 1 --2- 172.21.15.184:0/1900081131 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3b5802ff10 0x7f3b580323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3b640096a0 tx=0x7f3b64006b40).stop 2024-02-19T22:14:30.276 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.275+0000 7f3b74bae700 1 -- 172.21.15.184:0/1900081131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b700f6510 msgr2=0x7f3b70108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:30.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.275+0000 7f3b74bae700 1 --2- 172.21.15.184:0/1900081131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b700f6510 0x7f3b70108a00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f3b60005010 tx=0x7f3b6000b040).stop 2024-02-19T22:14:30.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.275+0000 7f3b74bae700 1 -- 172.21.15.184:0/1900081131 shutdown_connections 2024-02-19T22:14:30.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.275+0000 7f3b74bae700 1 --2- 172.21.15.184:0/1900081131 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3b5802ff10 0x7f3b580323d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:30.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.275+0000 7f3b74bae700 1 --2- 172.21.15.184:0/1900081131 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b700f6510 0x7f3b70108a00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:30.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.275+0000 7f3b74bae700 1 -- 172.21.15.184:0/1900081131 >> 172.21.15.184:0/1900081131 conn(0x7f3b700f1c70 msgr2=0x7f3b700f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:30.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.275+0000 7f3b74bae700 1 -- 172.21.15.184:0/1900081131 shutdown_connections 2024-02-19T22:14:30.277 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:30.275+0000 7f3b74bae700 1 -- 172.21.15.184:0/1900081131 wait complete. 2024-02-19T22:14:30.277 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:31.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:30 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:30.274449+0000 mon.smithi176 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.184:0/1900081131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:32.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:32.013 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:32.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:29.883207+0000 mgr.smithi176.faoogw (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:33.792 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.786+0000 7f59bda62700 1 -- 172.21.15.184:0/3122747540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b80f5790 msgr2=0x7f59b80f5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:33.792 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.786+0000 7f59bda62700 1 --2- 172.21.15.184:0/3122747540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b80f5790 0x7f59b80f5bb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f59ac009fb0 tx=0x7f59ac009b30).stop 2024-02-19T22:14:33.792 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.786+0000 7f59bda62700 1 -- 172.21.15.184:0/3122747540 shutdown_connections 2024-02-19T22:14:33.792 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.786+0000 7f59bda62700 1 --2- 172.21.15.184:0/3122747540 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b80f5790 0x7f59b80f5bb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:33.792 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.786+0000 7f59bda62700 1 -- 172.21.15.184:0/3122747540 >> 172.21.15.184:0/3122747540 conn(0x7f59b80f0120 msgr2=0x7f59b80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:33.792 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.786+0000 7f59bda62700 1 -- 172.21.15.184:0/3122747540 shutdown_connections 2024-02-19T22:14:33.792 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.786+0000 7f59bda62700 1 -- 172.21.15.184:0/3122747540 wait complete. 2024-02-19T22:14:33.792 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.786+0000 7f59bda62700 1 Processor -- start 2024-02-19T22:14:33.792 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.786+0000 7f59bda62700 1 -- start start 2024-02-19T22:14:33.793 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.787+0000 7f59bda62700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b80f5790 0x7f59b8106660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:33.793 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.787+0000 7f59bda62700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b8106b50 con 0x7f59b80f5790 2024-02-19T22:14:33.793 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.787+0000 7f59bca60700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b80f5790 0x7f59b8106660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:33.793 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.787+0000 7f59bca60700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b80f5790 0x7f59b8106660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:36142/0 (socket says 172.21.15.184:36142) 2024-02-19T22:14:33.793 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.787+0000 7f59bca60700 1 -- 172.21.15.184:0/3402122560 learned_addr learned my addr 172.21.15.184:0/3402122560 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:33.793 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.787+0000 7f59bca60700 1 -- 172.21.15.184:0/3402122560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ac005040 con 0x7f59b80f5790 2024-02-19T22:14:33.793 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.787+0000 7f59bca60700 1 --2- 172.21.15.184:0/3402122560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b80f5790 0x7f59b8106660 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f59ac009f80 tx=0x7f59ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:33.793 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.788+0000 7f59b57fa700 1 -- 172.21.15.184:0/3402122560 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59ac0095f0 con 0x7f59b80f5790 2024-02-19T22:14:33.793 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.788+0000 7f59bda62700 1 -- 172.21.15.184:0/3402122560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b8106d50 con 0x7f59b80f5790 2024-02-19T22:14:33.793 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.788+0000 7f59b57fa700 1 -- 172.21.15.184:0/3402122560 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59ac007e90 con 0x7f59b80f5790 2024-02-19T22:14:33.793 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.788+0000 7f59b57fa700 1 -- 172.21.15.184:0/3402122560 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59ac014e60 con 0x7f59b80f5790 2024-02-19T22:14:33.793 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.788+0000 7f59bda62700 1 -- 172.21.15.184:0/3402122560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b81071e0 con 0x7f59b80f5790 2024-02-19T22:14:33.793 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.788+0000 7f59b57fa700 1 -- 172.21.15.184:0/3402122560 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59ac025070 con 0x7f59b80f5790 2024-02-19T22:14:33.794 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.788+0000 7f59bda62700 1 -- 172.21.15.184:0/3402122560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b8045ad0 con 0x7f59b80f5790 2024-02-19T22:14:33.794 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.789+0000 7f59b57fa700 1 --2- 172.21.15.184:0/3402122560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f59a802ff10 0x7f59a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:33.794 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.789+0000 7f59b57fa700 1 -- 172.21.15.184:0/3402122560 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59ac0409a0 con 0x7f59b80f5790 2024-02-19T22:14:33.794 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.789+0000 7f59b7fff700 1 --2- 172.21.15.184:0/3402122560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f59a802ff10 0x7f59a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:33.794 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.789+0000 7f59b7fff700 1 --2- 172.21.15.184:0/3402122560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f59a802ff10 0x7f59a80323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f59a40096a0 tx=0x7f59a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:33.795 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:33.793+0000 7f59b57fa700 1 -- 172.21.15.184:0/3402122560 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59ac00f4e0 con 0x7f59b80f5790 2024-02-19T22:14:34.004 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:34.003+0000 7f59bda62700 1 -- 172.21.15.184:0/3402122560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59b8108090 con 0x7f59b80f5790 2024-02-19T22:14:34.005 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:34.003+0000 7f59b57fa700 1 -- 172.21.15.184:0/3402122560 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f59ac00f300 con 0x7f59b80f5790 2024-02-19T22:14:34.005 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:34.005 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:34.007 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:34.005+0000 7f59bda62700 1 -- 172.21.15.184:0/3402122560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f59a802ff10 msgr2=0x7f59a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:34.007 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:34.005+0000 7f59bda62700 1 --2- 172.21.15.184:0/3402122560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f59a802ff10 0x7f59a80323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f59a40096a0 tx=0x7f59a4006b40).stop 2024-02-19T22:14:34.007 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:34.005+0000 7f59bda62700 1 -- 172.21.15.184:0/3402122560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b80f5790 msgr2=0x7f59b8106660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:34.008 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:34.005+0000 7f59bda62700 1 --2- 172.21.15.184:0/3402122560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b80f5790 0x7f59b8106660 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f59ac009f80 tx=0x7f59ac00b040).stop 2024-02-19T22:14:34.008 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:34.005+0000 7f59bda62700 1 -- 172.21.15.184:0/3402122560 shutdown_connections 2024-02-19T22:14:34.008 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:34.005+0000 7f59bda62700 1 --2- 172.21.15.184:0/3402122560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f59a802ff10 0x7f59a80323d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:34.008 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:34.005+0000 7f59bda62700 1 --2- 172.21.15.184:0/3402122560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f59b80f5790 0x7f59b8106660 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:34.008 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:34.005+0000 7f59bda62700 1 -- 172.21.15.184:0/3402122560 >> 172.21.15.184:0/3402122560 conn(0x7f59b80f0120 msgr2=0x7f59b80f4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:34.008 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:34.005+0000 7f59bda62700 1 -- 172.21.15.184:0/3402122560 shutdown_connections 2024-02-19T22:14:34.009 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:34.005+0000 7f59bda62700 1 -- 172.21.15.184:0/3402122560 wait complete. 2024-02-19T22:14:34.009 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:34.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:31.883553+0000 mgr.smithi176.faoogw (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:35.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:34 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:34.004619+0000 mon.smithi176 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.184:0/3402122560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:35.657 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:35.657 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:36.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:33.883884+0000 mgr.smithi176.faoogw (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:35.971693+0000 mon.smithi176 (mon.0) 223 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:14:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:35.973939+0000 mon.smithi176 (mon.0) 224 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:14:37.561 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.558+0000 7f27dc148700 1 -- 172.21.15.184:0/3670499725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27d40f47c0 msgr2=0x7f27d40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:37.561 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.558+0000 7f27dc148700 1 --2- 172.21.15.184:0/3670499725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27d40f47c0 0x7f27d40f4be0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f27c4009fb0 tx=0x7f27c4009b30).stop 2024-02-19T22:14:37.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.558+0000 7f27dc148700 1 -- 172.21.15.184:0/3670499725 shutdown_connections 2024-02-19T22:14:37.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.558+0000 7f27dc148700 1 --2- 172.21.15.184:0/3670499725 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27d40f47c0 0x7f27d40f4be0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:37.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.558+0000 7f27dc148700 1 -- 172.21.15.184:0/3670499725 >> 172.21.15.184:0/3670499725 conn(0x7f27d40f02e0 msgr2=0x7f27d40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:37.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.559+0000 7f27dc148700 1 -- 172.21.15.184:0/3670499725 shutdown_connections 2024-02-19T22:14:37.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.559+0000 7f27dc148700 1 -- 172.21.15.184:0/3670499725 wait complete. 2024-02-19T22:14:37.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.559+0000 7f27dc148700 1 Processor -- start 2024-02-19T22:14:37.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.559+0000 7f27dc148700 1 -- start start 2024-02-19T22:14:37.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.559+0000 7f27dc148700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27d40f47c0 0x7f27d40f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:37.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.559+0000 7f27dc148700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d40f57f0 con 0x7f27d40f47c0 2024-02-19T22:14:37.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.559+0000 7f27d9ee4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27d40f47c0 0x7f27d40f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:37.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.559+0000 7f27d9ee4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27d40f47c0 0x7f27d40f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:32828/0 (socket says 172.21.15.184:32828) 2024-02-19T22:14:37.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.559+0000 7f27d9ee4700 1 -- 172.21.15.184:0/3609933271 learned_addr learned my addr 172.21.15.184:0/3609933271 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:37.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.560+0000 7f27d9ee4700 1 -- 172.21.15.184:0/3609933271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c4005040 con 0x7f27d40f47c0 2024-02-19T22:14:37.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.560+0000 7f27d9ee4700 1 --2- 172.21.15.184:0/3609933271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27d40f47c0 0x7f27d40f7370 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f27c4005010 tx=0x7f27c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:37.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.560+0000 7f27ca7fc700 1 -- 172.21.15.184:0/3609933271 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27c4009320 con 0x7f27d40f47c0 2024-02-19T22:14:37.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.560+0000 7f27dc148700 1 -- 172.21.15.184:0/3609933271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d40f5990 con 0x7f27d40f47c0 2024-02-19T22:14:37.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.560+0000 7f27ca7fc700 1 -- 172.21.15.184:0/3609933271 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27c4013380 con 0x7f27d40f47c0 2024-02-19T22:14:37.563 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.560+0000 7f27ca7fc700 1 -- 172.21.15.184:0/3609933271 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27c4014e50 con 0x7f27d40f47c0 2024-02-19T22:14:37.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.560+0000 7f27dc148700 1 -- 172.21.15.184:0/3609933271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d40f5e20 con 0x7f27d40f47c0 2024-02-19T22:14:37.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.561+0000 7f27ca7fc700 1 -- 172.21.15.184:0/3609933271 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27c40095f0 con 0x7f27d40f47c0 2024-02-19T22:14:37.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.561+0000 7f27dc148700 1 -- 172.21.15.184:0/3609933271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d4059100 con 0x7f27d40f47c0 2024-02-19T22:14:37.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.561+0000 7f27ca7fc700 1 --2- 172.21.15.184:0/3609933271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f27c002ff10 0x7f27c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:37.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.561+0000 7f27ca7fc700 1 -- 172.21.15.184:0/3609933271 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27c4040e90 con 0x7f27d40f47c0 2024-02-19T22:14:37.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.561+0000 7f27d96e3700 1 --2- 172.21.15.184:0/3609933271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f27c002ff10 0x7f27c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:37.564 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.562+0000 7f27d96e3700 1 --2- 172.21.15.184:0/3609933271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f27c002ff10 0x7f27c00323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f27d00096a0 tx=0x7f27d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:37.567 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.566+0000 7f27ca7fc700 1 -- 172.21.15.184:0/3609933271 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27c4014420 con 0x7f27d40f47c0 2024-02-19T22:14:37.774 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.772+0000 7f27dc148700 1 -- 172.21.15.184:0/3609933271 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27d40fc090 con 0x7f27d40f47c0 2024-02-19T22:14:37.775 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.773+0000 7f27ca7fc700 1 -- 172.21.15.184:0/3609933271 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f27c4010230 con 0x7f27d40f47c0 2024-02-19T22:14:37.775 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:37.775 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:37.776 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.775+0000 7f27dc148700 1 -- 172.21.15.184:0/3609933271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f27c002ff10 msgr2=0x7f27c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:37.776 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.775+0000 7f27dc148700 1 --2- 172.21.15.184:0/3609933271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f27c002ff10 0x7f27c00323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f27d00096a0 tx=0x7f27d0006b40).stop 2024-02-19T22:14:37.776 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.775+0000 7f27dc148700 1 -- 172.21.15.184:0/3609933271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27d40f47c0 msgr2=0x7f27d40f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:37.777 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.775+0000 7f27dc148700 1 --2- 172.21.15.184:0/3609933271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27d40f47c0 0x7f27d40f7370 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f27c4005010 tx=0x7f27c400b040).stop 2024-02-19T22:14:37.777 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.775+0000 7f27dc148700 1 -- 172.21.15.184:0/3609933271 shutdown_connections 2024-02-19T22:14:37.777 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.775+0000 7f27dc148700 1 --2- 172.21.15.184:0/3609933271 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f27c002ff10 0x7f27c00323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:37.777 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.775+0000 7f27dc148700 1 --2- 172.21.15.184:0/3609933271 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f27d40f47c0 0x7f27d40f7370 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:37.777 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.775+0000 7f27dc148700 1 -- 172.21.15.184:0/3609933271 >> 172.21.15.184:0/3609933271 conn(0x7f27d40f02e0 msgr2=0x7f27d40f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:37.777 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.775+0000 7f27dc148700 1 -- 172.21.15.184:0/3609933271 shutdown_connections 2024-02-19T22:14:37.777 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:37.775+0000 7f27dc148700 1 -- 172.21.15.184:0/3609933271 wait complete. 2024-02-19T22:14:37.777 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:38.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:35.884253+0000 mgr.smithi176.faoogw (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:38.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:37.774557+0000 mon.smithi176 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.184:0/3609933271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:39.392 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:39.392 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:40.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:37.884643+0000 mgr.smithi176.faoogw (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:41.202 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.200+0000 7f2459da2700 1 -- 172.21.15.184:0/1868337433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2454060280 msgr2=0x7f24540f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:41.202 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.200+0000 7f2459da2700 1 --2- 172.21.15.184:0/1868337433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2454060280 0x7f24540f90f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f243c009fb0 tx=0x7f243c009b30).stop 2024-02-19T22:14:41.202 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.200+0000 7f2459da2700 1 -- 172.21.15.184:0/1868337433 shutdown_connections 2024-02-19T22:14:41.202 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.200+0000 7f2459da2700 1 --2- 172.21.15.184:0/1868337433 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2454060280 0x7f24540f90f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:41.202 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.200+0000 7f2459da2700 1 -- 172.21.15.184:0/1868337433 >> 172.21.15.184:0/1868337433 conn(0x7f24540f0b70 msgr2=0x7f24540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:41.202 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.200+0000 7f2459da2700 1 -- 172.21.15.184:0/1868337433 shutdown_connections 2024-02-19T22:14:41.203 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.200+0000 7f2459da2700 1 -- 172.21.15.184:0/1868337433 wait complete. 2024-02-19T22:14:41.203 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.200+0000 7f2459da2700 1 Processor -- start 2024-02-19T22:14:41.203 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.200+0000 7f2459da2700 1 -- start start 2024-02-19T22:14:41.203 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.200+0000 7f2459da2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2454060280 0x7f2454106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:41.203 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.200+0000 7f2459da2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2454106c70 con 0x7f2454060280 2024-02-19T22:14:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.201+0000 7f24537fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2454060280 0x7f2454106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.201+0000 7f24537fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2454060280 0x7f2454106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:32850/0 (socket says 172.21.15.184:32850) 2024-02-19T22:14:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.201+0000 7f24537fe700 1 -- 172.21.15.184:0/3263736343 learned_addr learned my addr 172.21.15.184:0/3263736343 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.201+0000 7f24537fe700 1 -- 172.21.15.184:0/3263736343 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243c005040 con 0x7f2454060280 2024-02-19T22:14:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.201+0000 7f24537fe700 1 --2- 172.21.15.184:0/3263736343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2454060280 0x7f2454106780 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f243c009830 tx=0x7f243c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.201+0000 7f244bfff700 1 -- 172.21.15.184:0/3263736343 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f243c009320 con 0x7f2454060280 2024-02-19T22:14:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.201+0000 7f2459da2700 1 -- 172.21.15.184:0/3263736343 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2454106e70 con 0x7f2454060280 2024-02-19T22:14:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.201+0000 7f244bfff700 1 -- 172.21.15.184:0/3263736343 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f243c013380 con 0x7f2454060280 2024-02-19T22:14:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.201+0000 7f244bfff700 1 -- 172.21.15.184:0/3263736343 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f243c014420 con 0x7f2454060280 2024-02-19T22:14:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.201+0000 7f2459da2700 1 -- 172.21.15.184:0/3263736343 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2454107300 con 0x7f2454060280 2024-02-19T22:14:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.202+0000 7f244bfff700 1 -- 172.21.15.184:0/3263736343 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f243c014580 con 0x7f2454060280 2024-02-19T22:14:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.202+0000 7f2459da2700 1 -- 172.21.15.184:0/3263736343 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2454045ad0 con 0x7f2454060280 2024-02-19T22:14:41.205 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.202+0000 7f244bfff700 1 --2- 172.21.15.184:0/3263736343 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f244002ffb0 0x7f2440032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:41.206 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.202+0000 7f244bfff700 1 -- 172.21.15.184:0/3263736343 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f243c0410f0 con 0x7f2454060280 2024-02-19T22:14:41.206 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.203+0000 7f2452ffd700 1 --2- 172.21.15.184:0/3263736343 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f244002ffb0 0x7f2440032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:41.206 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.203+0000 7f2452ffd700 1 --2- 172.21.15.184:0/3263736343 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f244002ffb0 0x7f2440032470 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f24440096a0 tx=0x7f2444006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:41.209 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.207+0000 7f244bfff700 1 -- 172.21.15.184:0/3263736343 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f243c01c320 con 0x7f2454060280 2024-02-19T22:14:41.411 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.409+0000 7f2459da2700 1 -- 172.21.15.184:0/3263736343 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24540fc090 con 0x7f2454060280 2024-02-19T22:14:41.412 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.410+0000 7f244bfff700 1 -- 172.21.15.184:0/3263736343 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f243c02c2d0 con 0x7f2454060280 2024-02-19T22:14:41.412 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:41.412 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:41.413 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.412+0000 7f2459da2700 1 -- 172.21.15.184:0/3263736343 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f244002ffb0 msgr2=0x7f2440032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:41.414 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.412+0000 7f2459da2700 1 --2- 172.21.15.184:0/3263736343 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f244002ffb0 0x7f2440032470 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f24440096a0 tx=0x7f2444006b40).stop 2024-02-19T22:14:41.414 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.412+0000 7f2459da2700 1 -- 172.21.15.184:0/3263736343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2454060280 msgr2=0x7f2454106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:41.414 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.412+0000 7f2459da2700 1 --2- 172.21.15.184:0/3263736343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2454060280 0x7f2454106780 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f243c009830 tx=0x7f243c00b040).stop 2024-02-19T22:14:41.414 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.412+0000 7f2459da2700 1 -- 172.21.15.184:0/3263736343 shutdown_connections 2024-02-19T22:14:41.414 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.412+0000 7f2459da2700 1 --2- 172.21.15.184:0/3263736343 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f244002ffb0 0x7f2440032470 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:41.414 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.412+0000 7f2459da2700 1 --2- 172.21.15.184:0/3263736343 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2454060280 0x7f2454106780 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:41.414 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.412+0000 7f2459da2700 1 -- 172.21.15.184:0/3263736343 >> 172.21.15.184:0/3263736343 conn(0x7f24540f0b70 msgr2=0x7f24540f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:41.414 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.412+0000 7f2459da2700 1 -- 172.21.15.184:0/3263736343 shutdown_connections 2024-02-19T22:14:41.414 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:41.412+0000 7f2459da2700 1 -- 172.21.15.184:0/3263736343 wait complete. 2024-02-19T22:14:41.415 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:42.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:39.885015+0000 mgr.smithi176.faoogw (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:42.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:41.411475+0000 mon.smithi176 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.184:0/3263736343' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:43.162 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:43.163 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:44.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:41.885378+0000 mgr.smithi176.faoogw (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:45.061 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.058+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2667785831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f540f62a0 msgr2=0x7f3f540f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:45.061 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.058+0000 7f3f5b20c700 1 --2- 172.21.15.184:0/2667785831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f540f62a0 0x7f3f540f66c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f3f50009fb0 tx=0x7f3f50009b30).stop 2024-02-19T22:14:45.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.058+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2667785831 shutdown_connections 2024-02-19T22:14:45.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.058+0000 7f3f5b20c700 1 --2- 172.21.15.184:0/2667785831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f540f62a0 0x7f3f540f66c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:45.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.058+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2667785831 >> 172.21.15.184:0/2667785831 conn(0x7f3f540f1aa0 msgr2=0x7f3f540f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:45.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.058+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2667785831 shutdown_connections 2024-02-19T22:14:45.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.058+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2667785831 wait complete. 2024-02-19T22:14:45.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.058+0000 7f3f5b20c700 1 Processor -- start 2024-02-19T22:14:45.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.058+0000 7f3f5b20c700 1 -- start start 2024-02-19T22:14:45.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.059+0000 7f3f5b20c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f540f62a0 0x7f3f5406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:45.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.059+0000 7f3f5b20c700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f54069e20 con 0x7f3f540f62a0 2024-02-19T22:14:45.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.059+0000 7f3f5a20a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f540f62a0 0x7f3f5406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:45.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.059+0000 7f3f5a20a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f540f62a0 0x7f3f5406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:32864/0 (socket says 172.21.15.184:32864) 2024-02-19T22:14:45.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.059+0000 7f3f5a20a700 1 -- 172.21.15.184:0/2294459414 learned_addr learned my addr 172.21.15.184:0/2294459414 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:45.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.059+0000 7f3f5a20a700 1 -- 172.21.15.184:0/2294459414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f50005040 con 0x7f3f540f62a0 2024-02-19T22:14:45.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.059+0000 7f3f5a20a700 1 --2- 172.21.15.184:0/2294459414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f540f62a0 0x7f3f5406b9a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f3f50009830 tx=0x7f3f5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:45.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.059+0000 7f3f4affd700 1 -- 172.21.15.184:0/2294459414 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f50009320 con 0x7f3f540f62a0 2024-02-19T22:14:45.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.059+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2294459414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f54069fc0 con 0x7f3f540f62a0 2024-02-19T22:14:45.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.059+0000 7f3f4affd700 1 -- 172.21.15.184:0/2294459414 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f50013380 con 0x7f3f540f62a0 2024-02-19T22:14:45.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.059+0000 7f3f4affd700 1 -- 172.21.15.184:0/2294459414 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f50014420 con 0x7f3f540f62a0 2024-02-19T22:14:45.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.059+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2294459414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5406a450 con 0x7f3f540f62a0 2024-02-19T22:14:45.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.060+0000 7f3f4affd700 1 -- 172.21.15.184:0/2294459414 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f5001e420 con 0x7f3f540f62a0 2024-02-19T22:14:45.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.060+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2294459414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f54059100 con 0x7f3f540f62a0 2024-02-19T22:14:45.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.060+0000 7f3f4affd700 1 --2- 172.21.15.184:0/2294459414 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3f4002ff60 0x7f3f40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:45.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.060+0000 7f3f4affd700 1 -- 172.21.15.184:0/2294459414 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f50040eb0 con 0x7f3f540f62a0 2024-02-19T22:14:45.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.061+0000 7f3f59a09700 1 --2- 172.21.15.184:0/2294459414 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3f4002ff60 0x7f3f40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:45.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.061+0000 7f3f59a09700 1 --2- 172.21.15.184:0/2294459414 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3f4002ff60 0x7f3f40032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f3f440096a0 tx=0x7f3f44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:45.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.065+0000 7f3f4affd700 1 -- 172.21.15.184:0/2294459414 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f5001cb10 con 0x7f3f540f62a0 2024-02-19T22:14:45.271 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.269+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2294459414 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f54045ad0 con 0x7f3f540f62a0 2024-02-19T22:14:45.272 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.270+0000 7f3f4affd700 1 -- 172.21.15.184:0/2294459414 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3f50004490 con 0x7f3f540f62a0 2024-02-19T22:14:45.272 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:45.272 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:45.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.272+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2294459414 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3f4002ff60 msgr2=0x7f3f40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:45.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.272+0000 7f3f5b20c700 1 --2- 172.21.15.184:0/2294459414 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3f4002ff60 0x7f3f40032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f3f440096a0 tx=0x7f3f44006b40).stop 2024-02-19T22:14:45.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.272+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2294459414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f540f62a0 msgr2=0x7f3f5406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:45.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.272+0000 7f3f5b20c700 1 --2- 172.21.15.184:0/2294459414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f540f62a0 0x7f3f5406b9a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f3f50009830 tx=0x7f3f5000b040).stop 2024-02-19T22:14:45.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.272+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2294459414 shutdown_connections 2024-02-19T22:14:45.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.272+0000 7f3f5b20c700 1 --2- 172.21.15.184:0/2294459414 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3f4002ff60 0x7f3f40032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:45.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.272+0000 7f3f5b20c700 1 --2- 172.21.15.184:0/2294459414 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f540f62a0 0x7f3f5406b9a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:45.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.272+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2294459414 >> 172.21.15.184:0/2294459414 conn(0x7f3f540f1aa0 msgr2=0x7f3f540f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:45.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.272+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2294459414 shutdown_connections 2024-02-19T22:14:45.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:45.272+0000 7f3f5b20c700 1 -- 172.21.15.184:0/2294459414 wait complete. 2024-02-19T22:14:45.275 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:46.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:43.885687+0000 mgr.smithi176.faoogw (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:46.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:45.271575+0000 mon.smithi176 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.184:0/2294459414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:46.965 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:46.965 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:48.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:45.886044+0000 mgr.smithi176.faoogw (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:48.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.891+0000 7f1e417a7700 1 -- 172.21.15.184:0/915566816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e3c0f47b0 msgr2=0x7f1e3c0f4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:48.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.891+0000 7f1e417a7700 1 --2- 172.21.15.184:0/915566816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e3c0f47b0 0x7f1e3c0f4bd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f1e30009fb0 tx=0x7f1e30009b30).stop 2024-02-19T22:14:48.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.891+0000 7f1e417a7700 1 -- 172.21.15.184:0/915566816 shutdown_connections 2024-02-19T22:14:48.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.891+0000 7f1e417a7700 1 --2- 172.21.15.184:0/915566816 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e3c0f47b0 0x7f1e3c0f4bd0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:48.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.891+0000 7f1e417a7700 1 -- 172.21.15.184:0/915566816 >> 172.21.15.184:0/915566816 conn(0x7f1e3c0f02b0 msgr2=0x7f1e3c0f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:48.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.891+0000 7f1e417a7700 1 -- 172.21.15.184:0/915566816 shutdown_connections 2024-02-19T22:14:48.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.891+0000 7f1e417a7700 1 -- 172.21.15.184:0/915566816 wait complete. 2024-02-19T22:14:48.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.892+0000 7f1e417a7700 1 Processor -- start 2024-02-19T22:14:48.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.892+0000 7f1e417a7700 1 -- start start 2024-02-19T22:14:48.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.892+0000 7f1e417a7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e3c0f47b0 0x7f1e3c106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:48.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.892+0000 7f1e417a7700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c106c80 con 0x7f1e3c0f47b0 2024-02-19T22:14:48.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.892+0000 7f1e3affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e3c0f47b0 0x7f1e3c106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:48.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.893+0000 7f1e3affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e3c0f47b0 0x7f1e3c106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:45286/0 (socket says 172.21.15.184:45286) 2024-02-19T22:14:48.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.893+0000 7f1e3affd700 1 -- 172.21.15.184:0/3041451439 learned_addr learned my addr 172.21.15.184:0/3041451439 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:48.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.893+0000 7f1e3affd700 1 -- 172.21.15.184:0/3041451439 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e30005040 con 0x7f1e3c0f47b0 2024-02-19T22:14:48.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.893+0000 7f1e3affd700 1 --2- 172.21.15.184:0/3041451439 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e3c0f47b0 0x7f1e3c106790 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f1e30005010 tx=0x7f1e3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:48.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.893+0000 7f1e237fe700 1 -- 172.21.15.184:0/3041451439 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e30009320 con 0x7f1e3c0f47b0 2024-02-19T22:14:48.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.893+0000 7f1e237fe700 1 -- 172.21.15.184:0/3041451439 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e30013380 con 0x7f1e3c0f47b0 2024-02-19T22:14:48.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.893+0000 7f1e417a7700 1 -- 172.21.15.184:0/3041451439 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e3c106e80 con 0x7f1e3c0f47b0 2024-02-19T22:14:48.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.893+0000 7f1e237fe700 1 -- 172.21.15.184:0/3041451439 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e30014e50 con 0x7f1e3c0f47b0 2024-02-19T22:14:48.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.893+0000 7f1e417a7700 1 -- 172.21.15.184:0/3041451439 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e3c107310 con 0x7f1e3c0f47b0 2024-02-19T22:14:48.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.894+0000 7f1e237fe700 1 -- 172.21.15.184:0/3041451439 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e300263a0 con 0x7f1e3c0f47b0 2024-02-19T22:14:48.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.894+0000 7f1e237fe700 1 --2- 172.21.15.184:0/3041451439 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f1e2402ff10 0x7f1e240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:48.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.894+0000 7f1e237fe700 1 -- 172.21.15.184:0/3041451439 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e30040e80 con 0x7f1e3c0f47b0 2024-02-19T22:14:48.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.894+0000 7f1e417a7700 1 -- 172.21.15.184:0/3041451439 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3c045ad0 con 0x7f1e3c0f47b0 2024-02-19T22:14:48.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.894+0000 7f1e3a7fc700 1 --2- 172.21.15.184:0/3041451439 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f1e2402ff10 0x7f1e240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:48.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.895+0000 7f1e3a7fc700 1 --2- 172.21.15.184:0/3041451439 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f1e2402ff10 0x7f1e240323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f1e2c0096a0 tx=0x7f1e2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:48.900 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:48.899+0000 7f1e237fe700 1 -- 172.21.15.184:0/3041451439 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e30012920 con 0x7f1e3c0f47b0 2024-02-19T22:14:49.109 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:49.108+0000 7f1e417a7700 1 -- 172.21.15.184:0/3041451439 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e3c108030 con 0x7f1e3c0f47b0 2024-02-19T22:14:49.110 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:49.109+0000 7f1e237fe700 1 -- 172.21.15.184:0/3041451439 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1e3002b020 con 0x7f1e3c0f47b0 2024-02-19T22:14:49.110 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:49.110 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:49.112 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:49.110+0000 7f1e417a7700 1 -- 172.21.15.184:0/3041451439 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f1e2402ff10 msgr2=0x7f1e240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:49.112 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:49.110+0000 7f1e417a7700 1 --2- 172.21.15.184:0/3041451439 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f1e2402ff10 0x7f1e240323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f1e2c0096a0 tx=0x7f1e2c006b40).stop 2024-02-19T22:14:49.112 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:49.110+0000 7f1e417a7700 1 -- 172.21.15.184:0/3041451439 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e3c0f47b0 msgr2=0x7f1e3c106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:49.112 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:49.110+0000 7f1e417a7700 1 --2- 172.21.15.184:0/3041451439 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e3c0f47b0 0x7f1e3c106790 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f1e30005010 tx=0x7f1e3000b040).stop 2024-02-19T22:14:49.112 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:49.110+0000 7f1e417a7700 1 -- 172.21.15.184:0/3041451439 shutdown_connections 2024-02-19T22:14:49.112 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:49.110+0000 7f1e417a7700 1 --2- 172.21.15.184:0/3041451439 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f1e2402ff10 0x7f1e240323d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:49.112 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:49.110+0000 7f1e417a7700 1 --2- 172.21.15.184:0/3041451439 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1e3c0f47b0 0x7f1e3c106790 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:49.113 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:49.110+0000 7f1e417a7700 1 -- 172.21.15.184:0/3041451439 >> 172.21.15.184:0/3041451439 conn(0x7f1e3c0f02b0 msgr2=0x7f1e3c0f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:49.113 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:49.110+0000 7f1e417a7700 1 -- 172.21.15.184:0/3041451439 shutdown_connections 2024-02-19T22:14:49.113 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:49.110+0000 7f1e417a7700 1 -- 172.21.15.184:0/3041451439 wait complete. 2024-02-19T22:14:49.113 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:50.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:49 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:47.886378+0000 mgr.smithi176.faoogw (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:50.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:49 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:49.109912+0000 mon.smithi176 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.184:0/3041451439' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:50.884 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:50.884 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:52.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:51 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:49.886715+0000 mgr.smithi176.faoogw (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:52.837 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.834+0000 7fbd018ae700 1 -- 172.21.15.184:0/1707062071 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcfc0f47c0 msgr2=0x7fbcfc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:52.837 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.834+0000 7fbd018ae700 1 --2- 172.21.15.184:0/1707062071 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcfc0f47c0 0x7fbcfc0f4be0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fbcf0009fb0 tx=0x7fbcf0009b30).stop 2024-02-19T22:14:52.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.834+0000 7fbd018ae700 1 -- 172.21.15.184:0/1707062071 shutdown_connections 2024-02-19T22:14:52.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.834+0000 7fbd018ae700 1 --2- 172.21.15.184:0/1707062071 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcfc0f47c0 0x7fbcfc0f4be0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:52.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.834+0000 7fbd018ae700 1 -- 172.21.15.184:0/1707062071 >> 172.21.15.184:0/1707062071 conn(0x7fbcfc0f02e0 msgr2=0x7fbcfc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:52.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.834+0000 7fbd018ae700 1 -- 172.21.15.184:0/1707062071 shutdown_connections 2024-02-19T22:14:52.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.834+0000 7fbd018ae700 1 -- 172.21.15.184:0/1707062071 wait complete. 2024-02-19T22:14:52.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.834+0000 7fbd018ae700 1 Processor -- start 2024-02-19T22:14:52.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.834+0000 7fbd018ae700 1 -- start start 2024-02-19T22:14:52.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.834+0000 7fbd018ae700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcfc0f47c0 0x7fbcfc1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:52.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.834+0000 7fbd018ae700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc106cc0 con 0x7fbcfc0f47c0 2024-02-19T22:14:52.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.835+0000 7fbcfaffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcfc0f47c0 0x7fbcfc1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:52.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.835+0000 7fbcfaffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcfc0f47c0 0x7fbcfc1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:45298/0 (socket says 172.21.15.184:45298) 2024-02-19T22:14:52.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.835+0000 7fbcfaffd700 1 -- 172.21.15.184:0/1355585648 learned_addr learned my addr 172.21.15.184:0/1355585648 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:52.838 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.835+0000 7fbcfaffd700 1 -- 172.21.15.184:0/1355585648 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf0005040 con 0x7fbcfc0f47c0 2024-02-19T22:14:52.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.835+0000 7fbcfaffd700 1 --2- 172.21.15.184:0/1355585648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcfc0f47c0 0x7fbcfc1067d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fbcf0005010 tx=0x7fbcf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:52.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.835+0000 7fbce37fe700 1 -- 172.21.15.184:0/1355585648 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcf0009320 con 0x7fbcfc0f47c0 2024-02-19T22:14:52.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.835+0000 7fbd018ae700 1 -- 172.21.15.184:0/1355585648 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcfc106ec0 con 0x7fbcfc0f47c0 2024-02-19T22:14:52.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.835+0000 7fbce37fe700 1 -- 172.21.15.184:0/1355585648 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbcf0013380 con 0x7fbcfc0f47c0 2024-02-19T22:14:52.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.835+0000 7fbce37fe700 1 -- 172.21.15.184:0/1355585648 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcf0014e50 con 0x7fbcfc0f47c0 2024-02-19T22:14:52.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.835+0000 7fbd018ae700 1 -- 172.21.15.184:0/1355585648 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcfc107350 con 0x7fbcfc0f47c0 2024-02-19T22:14:52.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.836+0000 7fbce37fe700 1 -- 172.21.15.184:0/1355585648 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbcf00095f0 con 0x7fbcfc0f47c0 2024-02-19T22:14:52.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.836+0000 7fbd018ae700 1 -- 172.21.15.184:0/1355585648 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcfc045ad0 con 0x7fbcfc0f47c0 2024-02-19T22:14:52.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.836+0000 7fbce37fe700 1 --2- 172.21.15.184:0/1355585648 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbce402ff10 0x7fbce40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:52.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.836+0000 7fbce37fe700 1 -- 172.21.15.184:0/1355585648 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbcf0040e90 con 0x7fbcfc0f47c0 2024-02-19T22:14:52.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.837+0000 7fbcfa7fc700 1 --2- 172.21.15.184:0/1355585648 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbce402ff10 0x7fbce40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:52.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.837+0000 7fbcfa7fc700 1 --2- 172.21.15.184:0/1355585648 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbce402ff10 0x7fbce40323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fbcec0096a0 tx=0x7fbcec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:52.842 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:52.841+0000 7fbce37fe700 1 -- 172.21.15.184:0/1355585648 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcf0014420 con 0x7fbcfc0f47c0 2024-02-19T22:14:53.047 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:53.046+0000 7fbd018ae700 1 -- 172.21.15.184:0/1355585648 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbcfc108200 con 0x7fbcfc0f47c0 2024-02-19T22:14:53.048 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:53.046+0000 7fbce37fe700 1 -- 172.21.15.184:0/1355585648 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbcf0010230 con 0x7fbcfc0f47c0 2024-02-19T22:14:53.048 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:53.048 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:53.049 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:53.048+0000 7fbd018ae700 1 -- 172.21.15.184:0/1355585648 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbce402ff10 msgr2=0x7fbce40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:53.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:53.048+0000 7fbd018ae700 1 --2- 172.21.15.184:0/1355585648 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbce402ff10 0x7fbce40323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fbcec0096a0 tx=0x7fbcec006b40).stop 2024-02-19T22:14:53.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:53.048+0000 7fbd018ae700 1 -- 172.21.15.184:0/1355585648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcfc0f47c0 msgr2=0x7fbcfc1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:53.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:53.048+0000 7fbd018ae700 1 --2- 172.21.15.184:0/1355585648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcfc0f47c0 0x7fbcfc1067d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fbcf0005010 tx=0x7fbcf000b040).stop 2024-02-19T22:14:53.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:53.048+0000 7fbd018ae700 1 -- 172.21.15.184:0/1355585648 shutdown_connections 2024-02-19T22:14:53.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:53.048+0000 7fbd018ae700 1 --2- 172.21.15.184:0/1355585648 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbce402ff10 0x7fbce40323d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:53.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:53.048+0000 7fbd018ae700 1 --2- 172.21.15.184:0/1355585648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbcfc0f47c0 0x7fbcfc1067d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:53.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:53.048+0000 7fbd018ae700 1 -- 172.21.15.184:0/1355585648 >> 172.21.15.184:0/1355585648 conn(0x7fbcfc0f02e0 msgr2=0x7fbcfc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:53.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:53.048+0000 7fbd018ae700 1 -- 172.21.15.184:0/1355585648 shutdown_connections 2024-02-19T22:14:53.050 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:53.048+0000 7fbd018ae700 1 -- 172.21.15.184:0/1355585648 wait complete. 2024-02-19T22:14:53.051 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:54.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:51.886978+0000 mgr.smithi176.faoogw (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:54.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:53.047824+0000 mon.smithi176 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.184:0/1355585648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:54.726 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:54.727 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:56.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:55 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:53.887288+0000 mgr.smithi176.faoogw (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:56.627 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.613+0000 7f6130cf4700 1 -- 172.21.15.184:0/551925355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f612c0f51a0 msgr2=0x7f612c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:56.627 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.613+0000 7f6130cf4700 1 --2- 172.21.15.184:0/551925355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f612c0f51a0 0x7f612c0f55c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f611c009fb0 tx=0x7f611c009b30).stop 2024-02-19T22:14:56.627 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.613+0000 7f6130cf4700 1 -- 172.21.15.184:0/551925355 shutdown_connections 2024-02-19T22:14:56.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.613+0000 7f6130cf4700 1 --2- 172.21.15.184:0/551925355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f612c0f51a0 0x7f612c0f55c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:56.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.613+0000 7f6130cf4700 1 -- 172.21.15.184:0/551925355 >> 172.21.15.184:0/551925355 conn(0x7f612c0f09a0 msgr2=0x7f612c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:56.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.613+0000 7f6130cf4700 1 -- 172.21.15.184:0/551925355 shutdown_connections 2024-02-19T22:14:56.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.614+0000 7f6130cf4700 1 -- 172.21.15.184:0/551925355 wait complete. 2024-02-19T22:14:56.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.614+0000 7f6130cf4700 1 Processor -- start 2024-02-19T22:14:56.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.614+0000 7f6130cf4700 1 -- start start 2024-02-19T22:14:56.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.615+0000 7f6130cf4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f612c0f51a0 0x7f612c1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:56.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.615+0000 7f6130cf4700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612c1059c0 con 0x7f612c0f51a0 2024-02-19T22:14:56.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.615+0000 7f612b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f612c0f51a0 0x7f612c1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:56.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.615+0000 7f612b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f612c0f51a0 0x7f612c1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:45302/0 (socket says 172.21.15.184:45302) 2024-02-19T22:14:56.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.615+0000 7f612b7fe700 1 -- 172.21.15.184:0/1915416256 learned_addr learned my addr 172.21.15.184:0/1915416256 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:14:56.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.615+0000 7f612b7fe700 1 -- 172.21.15.184:0/1915416256 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c005040 con 0x7f612c0f51a0 2024-02-19T22:14:56.628 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.615+0000 7f612b7fe700 1 --2- 172.21.15.184:0/1915416256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f612c0f51a0 0x7f612c1054d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f611c0089f0 tx=0x7f611c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:56.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.615+0000 7f6113fff700 1 -- 172.21.15.184:0/1915416256 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f611c009320 con 0x7f612c0f51a0 2024-02-19T22:14:56.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.615+0000 7f6130cf4700 1 -- 172.21.15.184:0/1915416256 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f612c105bc0 con 0x7f612c0f51a0 2024-02-19T22:14:56.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.615+0000 7f6113fff700 1 -- 172.21.15.184:0/1915416256 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f611c013380 con 0x7f612c0f51a0 2024-02-19T22:14:56.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.615+0000 7f6113fff700 1 -- 172.21.15.184:0/1915416256 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f611c014e20 con 0x7f612c0f51a0 2024-02-19T22:14:56.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.615+0000 7f6130cf4700 1 -- 172.21.15.184:0/1915416256 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f612c106050 con 0x7f612c0f51a0 2024-02-19T22:14:56.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.616+0000 7f6113fff700 1 -- 172.21.15.184:0/1915416256 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f611c0095f0 con 0x7f612c0f51a0 2024-02-19T22:14:56.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.616+0000 7f6130cf4700 1 -- 172.21.15.184:0/1915416256 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f612c045ad0 con 0x7f612c0f51a0 2024-02-19T22:14:56.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.616+0000 7f6113fff700 1 --2- 172.21.15.184:0/1915416256 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f611402ff10 0x7f61140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:56.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.616+0000 7f6113fff700 1 -- 172.21.15.184:0/1915416256 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f611c040f10 con 0x7f612c0f51a0 2024-02-19T22:14:56.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.616+0000 7f612affd700 1 --2- 172.21.15.184:0/1915416256 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f611402ff10 0x7f61140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:56.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.617+0000 7f612affd700 1 --2- 172.21.15.184:0/1915416256 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f611402ff10 0x7f61140323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f61200096a0 tx=0x7f6120006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:56.629 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.621+0000 7f6113fff700 1 -- 172.21.15.184:0/1915416256 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f611c014420 con 0x7f612c0f51a0 2024-02-19T22:14:56.827 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.826+0000 7f6130cf4700 1 -- 172.21.15.184:0/1915416256 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f612c106f00 con 0x7f612c0f51a0 2024-02-19T22:14:56.828 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.826+0000 7f6113fff700 1 -- 172.21.15.184:0/1915416256 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f611c01d980 con 0x7f612c0f51a0 2024-02-19T22:14:56.828 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:14:56.828 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:56.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.828+0000 7f6130cf4700 1 -- 172.21.15.184:0/1915416256 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f611402ff10 msgr2=0x7f61140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:56.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.828+0000 7f6130cf4700 1 --2- 172.21.15.184:0/1915416256 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f611402ff10 0x7f61140323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f61200096a0 tx=0x7f6120006b40).stop 2024-02-19T22:14:56.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.828+0000 7f6130cf4700 1 -- 172.21.15.184:0/1915416256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f612c0f51a0 msgr2=0x7f612c1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:56.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.828+0000 7f6130cf4700 1 --2- 172.21.15.184:0/1915416256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f612c0f51a0 0x7f612c1054d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f611c0089f0 tx=0x7f611c00b040).stop 2024-02-19T22:14:56.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.828+0000 7f6130cf4700 1 -- 172.21.15.184:0/1915416256 shutdown_connections 2024-02-19T22:14:56.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.828+0000 7f6130cf4700 1 --2- 172.21.15.184:0/1915416256 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f611402ff10 0x7f61140323d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:56.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.828+0000 7f6130cf4700 1 --2- 172.21.15.184:0/1915416256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f612c0f51a0 0x7f612c1054d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:56.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.828+0000 7f6130cf4700 1 -- 172.21.15.184:0/1915416256 >> 172.21.15.184:0/1915416256 conn(0x7f612c0f09a0 msgr2=0x7f612c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:56.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.828+0000 7f6130cf4700 1 -- 172.21.15.184:0/1915416256 shutdown_connections 2024-02-19T22:14:56.830 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:14:56.828+0000 7f6130cf4700 1 -- 172.21.15.184:0/1915416256 wait complete. 2024-02-19T22:14:56.831 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:14:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:55.887647+0000 mgr.smithi176.faoogw (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:14:56.827569+0000 mon.smithi176 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.184:0/1915416256' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:58.548 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:58.548 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:59.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:14:58 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:57.887892+0000 mgr.smithi176.faoogw (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:00.385 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.383+0000 7fccba08b700 1 -- 172.21.15.184:0/3950620379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccb40f6510 msgr2=0x7fccb40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:00.385 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.383+0000 7fccba08b700 1 --2- 172.21.15.184:0/3950620379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccb40f6510 0x7fccb40f6930 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fcc9c009fb0 tx=0x7fcc9c009b30).stop 2024-02-19T22:15:00.385 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.383+0000 7fccba08b700 1 -- 172.21.15.184:0/3950620379 shutdown_connections 2024-02-19T22:15:00.385 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.383+0000 7fccba08b700 1 --2- 172.21.15.184:0/3950620379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccb40f6510 0x7fccb40f6930 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:00.385 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.383+0000 7fccba08b700 1 -- 172.21.15.184:0/3950620379 >> 172.21.15.184:0/3950620379 conn(0x7fccb40f1c70 msgr2=0x7fccb40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:00.385 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.383+0000 7fccba08b700 1 -- 172.21.15.184:0/3950620379 shutdown_connections 2024-02-19T22:15:00.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.383+0000 7fccba08b700 1 -- 172.21.15.184:0/3950620379 wait complete. 2024-02-19T22:15:00.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.383+0000 7fccba08b700 1 Processor -- start 2024-02-19T22:15:00.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.384+0000 7fccba08b700 1 -- start start 2024-02-19T22:15:00.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.384+0000 7fccba08b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccb40f6510 0x7fccb41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:00.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.384+0000 7fccba08b700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb4106cd0 con 0x7fccb40f6510 2024-02-19T22:15:00.387 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.384+0000 7fccb37fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccb40f6510 0x7fccb41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:00.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.384+0000 7fccb37fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccb40f6510 0x7fccb41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:45314/0 (socket says 172.21.15.184:45314) 2024-02-19T22:15:00.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.384+0000 7fccb37fe700 1 -- 172.21.15.184:0/2694821665 learned_addr learned my addr 172.21.15.184:0/2694821665 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:00.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.384+0000 7fccb37fe700 1 -- 172.21.15.184:0/2694821665 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c005040 con 0x7fccb40f6510 2024-02-19T22:15:00.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.384+0000 7fccb37fe700 1 --2- 172.21.15.184:0/2694821665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccb40f6510 0x7fccb41067e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fcc9c005010 tx=0x7fcc9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:00.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.384+0000 7fccabfff700 1 -- 172.21.15.184:0/2694821665 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc9c009320 con 0x7fccb40f6510 2024-02-19T22:15:00.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.384+0000 7fccba08b700 1 -- 172.21.15.184:0/2694821665 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb4106ed0 con 0x7fccb40f6510 2024-02-19T22:15:00.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.384+0000 7fccabfff700 1 -- 172.21.15.184:0/2694821665 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc9c013380 con 0x7fccb40f6510 2024-02-19T22:15:00.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.384+0000 7fccabfff700 1 -- 172.21.15.184:0/2694821665 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc9c014e50 con 0x7fccb40f6510 2024-02-19T22:15:00.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.384+0000 7fccba08b700 1 -- 172.21.15.184:0/2694821665 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb4107360 con 0x7fccb40f6510 2024-02-19T22:15:00.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.385+0000 7fccabfff700 1 -- 172.21.15.184:0/2694821665 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc9c0263a0 con 0x7fccb40f6510 2024-02-19T22:15:00.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.385+0000 7fccba08b700 1 -- 172.21.15.184:0/2694821665 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb4045ad0 con 0x7fccb40f6510 2024-02-19T22:15:00.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.385+0000 7fccabfff700 1 --2- 172.21.15.184:0/2694821665 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcca002ff10 0x7fcca00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:00.389 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.385+0000 7fccabfff700 1 -- 172.21.15.184:0/2694821665 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc9c040e90 con 0x7fccb40f6510 2024-02-19T22:15:00.389 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.386+0000 7fccb2ffd700 1 --2- 172.21.15.184:0/2694821665 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcca002ff10 0x7fcca00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:00.389 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.386+0000 7fccb2ffd700 1 --2- 172.21.15.184:0/2694821665 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcca002ff10 0x7fcca00323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fcca40096a0 tx=0x7fcca4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:00.391 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.390+0000 7fccabfff700 1 -- 172.21.15.184:0/2694821665 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc9c0126f0 con 0x7fccb40f6510 2024-02-19T22:15:00.598 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.596+0000 7fccba08b700 1 -- 172.21.15.184:0/2694821665 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fccb4108210 con 0x7fccb40f6510 2024-02-19T22:15:00.598 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.597+0000 7fccabfff700 1 -- 172.21.15.184:0/2694821665 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcc9c012330 con 0x7fccb40f6510 2024-02-19T22:15:00.599 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:00.599 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:00.600 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.598+0000 7fccba08b700 1 -- 172.21.15.184:0/2694821665 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcca002ff10 msgr2=0x7fcca00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:00.600 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.598+0000 7fccba08b700 1 --2- 172.21.15.184:0/2694821665 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcca002ff10 0x7fcca00323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fcca40096a0 tx=0x7fcca4006b40).stop 2024-02-19T22:15:00.600 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.598+0000 7fccba08b700 1 -- 172.21.15.184:0/2694821665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccb40f6510 msgr2=0x7fccb41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:00.600 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.598+0000 7fccba08b700 1 --2- 172.21.15.184:0/2694821665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccb40f6510 0x7fccb41067e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fcc9c005010 tx=0x7fcc9c00b040).stop 2024-02-19T22:15:00.601 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.598+0000 7fccba08b700 1 -- 172.21.15.184:0/2694821665 shutdown_connections 2024-02-19T22:15:00.601 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.599+0000 7fccba08b700 1 --2- 172.21.15.184:0/2694821665 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcca002ff10 0x7fcca00323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:00.601 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.599+0000 7fccba08b700 1 --2- 172.21.15.184:0/2694821665 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fccb40f6510 0x7fccb41067e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:00.601 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.599+0000 7fccba08b700 1 -- 172.21.15.184:0/2694821665 >> 172.21.15.184:0/2694821665 conn(0x7fccb40f1c70 msgr2=0x7fccb40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:00.601 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.599+0000 7fccba08b700 1 -- 172.21.15.184:0/2694821665 shutdown_connections 2024-02-19T22:15:00.601 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:00.599+0000 7fccba08b700 1 -- 172.21.15.184:0/2694821665 wait complete. 2024-02-19T22:15:00.601 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:01.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:00 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:00.598289+0000 mon.smithi176 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.184:0/2694821665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:02.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:14:59.888194+0000 mgr.smithi176.faoogw (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:02.404 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:02.405 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:04.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:01.888559+0000 mgr.smithi176.faoogw (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:04.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.285+0000 7fad9f30e700 1 -- 172.21.15.184:0/1711505447 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad980f5410 msgr2=0x7fad980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:04.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.285+0000 7fad9f30e700 1 --2- 172.21.15.184:0/1711505447 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad980f5410 0x7fad980f5830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fad88009fb0 tx=0x7fad88009b30).stop 2024-02-19T22:15:04.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.286+0000 7fad9f30e700 1 -- 172.21.15.184:0/1711505447 shutdown_connections 2024-02-19T22:15:04.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.286+0000 7fad9f30e700 1 --2- 172.21.15.184:0/1711505447 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad980f5410 0x7fad980f5830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:04.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.286+0000 7fad9f30e700 1 -- 172.21.15.184:0/1711505447 >> 172.21.15.184:0/1711505447 conn(0x7fad980f0b70 msgr2=0x7fad980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:04.288 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.286+0000 7fad9f30e700 1 -- 172.21.15.184:0/1711505447 shutdown_connections 2024-02-19T22:15:04.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.286+0000 7fad9f30e700 1 -- 172.21.15.184:0/1711505447 wait complete. 2024-02-19T22:15:04.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.286+0000 7fad9f30e700 1 Processor -- start 2024-02-19T22:15:04.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.286+0000 7fad9f30e700 1 -- start start 2024-02-19T22:15:04.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.286+0000 7fad9f30e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad980f5410 0x7fad981094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:04.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.286+0000 7fad9f30e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad98060240 con 0x7fad980f5410 2024-02-19T22:15:04.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.287+0000 7fad9d0aa700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad980f5410 0x7fad981094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:04.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.287+0000 7fad9d0aa700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad980f5410 0x7fad981094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:45328/0 (socket says 172.21.15.184:45328) 2024-02-19T22:15:04.289 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.287+0000 7fad9d0aa700 1 -- 172.21.15.184:0/2593685054 learned_addr learned my addr 172.21.15.184:0/2593685054 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:04.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.287+0000 7fad9d0aa700 1 -- 172.21.15.184:0/2593685054 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad88005040 con 0x7fad980f5410 2024-02-19T22:15:04.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.287+0000 7fad9d0aa700 1 --2- 172.21.15.184:0/2593685054 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad980f5410 0x7fad981094e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fad88005010 tx=0x7fad8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:04.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.287+0000 7fad8dffb700 1 -- 172.21.15.184:0/2593685054 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad88009320 con 0x7fad980f5410 2024-02-19T22:15:04.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.287+0000 7fad9f30e700 1 -- 172.21.15.184:0/2593685054 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad98060440 con 0x7fad980f5410 2024-02-19T22:15:04.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.287+0000 7fad8dffb700 1 -- 172.21.15.184:0/2593685054 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad88013380 con 0x7fad980f5410 2024-02-19T22:15:04.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.287+0000 7fad8dffb700 1 -- 172.21.15.184:0/2593685054 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad88014e50 con 0x7fad980f5410 2024-02-19T22:15:04.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.287+0000 7fad9f30e700 1 -- 172.21.15.184:0/2593685054 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad98109b90 con 0x7fad980f5410 2024-02-19T22:15:04.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.288+0000 7fad8dffb700 1 -- 172.21.15.184:0/2593685054 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad880263a0 con 0x7fad980f5410 2024-02-19T22:15:04.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.288+0000 7fad9f30e700 1 -- 172.21.15.184:0/2593685054 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad98045ad0 con 0x7fad980f5410 2024-02-19T22:15:04.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.288+0000 7fad8dffb700 1 --2- 172.21.15.184:0/2593685054 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fad8402ff10 0x7fad840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:04.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.288+0000 7fad8dffb700 1 -- 172.21.15.184:0/2593685054 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad88041260 con 0x7fad980f5410 2024-02-19T22:15:04.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.288+0000 7fad9c8a9700 1 --2- 172.21.15.184:0/2593685054 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fad8402ff10 0x7fad840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:04.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.289+0000 7fad9c8a9700 1 --2- 172.21.15.184:0/2593685054 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fad8402ff10 0x7fad840323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fad940096a0 tx=0x7fad94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:04.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.293+0000 7fad8dffb700 1 -- 172.21.15.184:0/2593685054 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad8801ccf0 con 0x7fad980f5410 2024-02-19T22:15:04.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.498+0000 7fad9f30e700 1 -- 172.21.15.184:0/2593685054 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad9810a3b0 con 0x7fad980f5410 2024-02-19T22:15:04.500 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.498+0000 7fad8dffb700 1 -- 172.21.15.184:0/2593685054 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fad8801cb10 con 0x7fad980f5410 2024-02-19T22:15:04.500 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:04.500 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:04.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.500+0000 7fad9f30e700 1 -- 172.21.15.184:0/2593685054 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fad8402ff10 msgr2=0x7fad840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:04.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.500+0000 7fad9f30e700 1 --2- 172.21.15.184:0/2593685054 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fad8402ff10 0x7fad840323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fad940096a0 tx=0x7fad94006b40).stop 2024-02-19T22:15:04.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.500+0000 7fad9f30e700 1 -- 172.21.15.184:0/2593685054 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad980f5410 msgr2=0x7fad981094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:04.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.500+0000 7fad9f30e700 1 --2- 172.21.15.184:0/2593685054 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad980f5410 0x7fad981094e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fad88005010 tx=0x7fad8800b040).stop 2024-02-19T22:15:04.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.500+0000 7fad9f30e700 1 -- 172.21.15.184:0/2593685054 shutdown_connections 2024-02-19T22:15:04.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.500+0000 7fad9f30e700 1 --2- 172.21.15.184:0/2593685054 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fad8402ff10 0x7fad840323d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:04.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.500+0000 7fad9f30e700 1 --2- 172.21.15.184:0/2593685054 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fad980f5410 0x7fad981094e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:04.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.500+0000 7fad9f30e700 1 -- 172.21.15.184:0/2593685054 >> 172.21.15.184:0/2593685054 conn(0x7fad980f0b70 msgr2=0x7fad980f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:04.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.500+0000 7fad9f30e700 1 -- 172.21.15.184:0/2593685054 shutdown_connections 2024-02-19T22:15:04.502 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:04.500+0000 7fad9f30e700 1 -- 172.21.15.184:0/2593685054 wait complete. 2024-02-19T22:15:04.503 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:05.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:04 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:04.499816+0000 mon.smithi176 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.184:0/2593685054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:06.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:03.888879+0000 mgr.smithi176.faoogw (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:06.183 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:06.184 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:08.070 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.067+0000 7f485f53a700 1 -- 172.21.15.184:0/3005568163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4858060280 msgr2=0x7f48580f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:08.070 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.067+0000 7f485f53a700 1 --2- 172.21.15.184:0/3005568163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4858060280 0x7f48580f90f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f4848009fb0 tx=0x7f4848009b30).stop 2024-02-19T22:15:08.070 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.067+0000 7f485f53a700 1 -- 172.21.15.184:0/3005568163 shutdown_connections 2024-02-19T22:15:08.070 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.067+0000 7f485f53a700 1 --2- 172.21.15.184:0/3005568163 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4858060280 0x7f48580f90f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:08.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.067+0000 7f485f53a700 1 -- 172.21.15.184:0/3005568163 >> 172.21.15.184:0/3005568163 conn(0x7f48580f0b70 msgr2=0x7f48580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:08.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.067+0000 7f485f53a700 1 -- 172.21.15.184:0/3005568163 shutdown_connections 2024-02-19T22:15:08.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.067+0000 7f485f53a700 1 -- 172.21.15.184:0/3005568163 wait complete. 2024-02-19T22:15:08.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.067+0000 7f485f53a700 1 Processor -- start 2024-02-19T22:15:08.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.067+0000 7f485f53a700 1 -- start start 2024-02-19T22:15:08.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.068+0000 7f485f53a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4858060280 0x7f4858106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:08.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.068+0000 7f485f53a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4858106c70 con 0x7f4858060280 2024-02-19T22:15:08.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.068+0000 7f485d2d6700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4858060280 0x7f4858106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:08.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.068+0000 7f485d2d6700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4858060280 0x7f4858106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:32918/0 (socket says 172.21.15.184:32918) 2024-02-19T22:15:08.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.068+0000 7f485d2d6700 1 -- 172.21.15.184:0/1442117961 learned_addr learned my addr 172.21.15.184:0/1442117961 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:08.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.068+0000 7f485d2d6700 1 -- 172.21.15.184:0/1442117961 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4848005040 con 0x7f4858060280 2024-02-19T22:15:08.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.068+0000 7f485d2d6700 1 --2- 172.21.15.184:0/1442117961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4858060280 0x7f4858106780 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f48480089f0 tx=0x7f484800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:08.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.068+0000 7f484dffb700 1 -- 172.21.15.184:0/1442117961 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4848009320 con 0x7f4858060280 2024-02-19T22:15:08.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.068+0000 7f484dffb700 1 -- 172.21.15.184:0/1442117961 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4848013380 con 0x7f4858060280 2024-02-19T22:15:08.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.068+0000 7f485f53a700 1 -- 172.21.15.184:0/1442117961 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4858106e70 con 0x7f4858060280 2024-02-19T22:15:08.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.068+0000 7f484dffb700 1 -- 172.21.15.184:0/1442117961 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4848014e20 con 0x7f4858060280 2024-02-19T22:15:08.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.069+0000 7f485f53a700 1 -- 172.21.15.184:0/1442117961 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4858107300 con 0x7f4858060280 2024-02-19T22:15:08.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.069+0000 7f484dffb700 1 -- 172.21.15.184:0/1442117961 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48480095f0 con 0x7f4858060280 2024-02-19T22:15:08.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.069+0000 7f485f53a700 1 -- 172.21.15.184:0/1442117961 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4858045ad0 con 0x7f4858060280 2024-02-19T22:15:08.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.070+0000 7f484dffb700 1 --2- 172.21.15.184:0/1442117961 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f484402ff10 0x7f48440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:08.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.070+0000 7f484dffb700 1 -- 172.21.15.184:0/1442117961 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4848040e90 con 0x7f4858060280 2024-02-19T22:15:08.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.070+0000 7f485cad5700 1 --2- 172.21.15.184:0/1442117961 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f484402ff10 0x7f48440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:08.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.070+0000 7f485cad5700 1 --2- 172.21.15.184:0/1442117961 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f484402ff10 0x7f48440323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f48540096a0 tx=0x7f4854006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:08.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.074+0000 7f484dffb700 1 -- 172.21.15.184:0/1442117961 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4848014420 con 0x7f4858060280 2024-02-19T22:15:08.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:05.889272+0000 mgr.smithi176.faoogw (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:08.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.282+0000 7f485f53a700 1 -- 172.21.15.184:0/1442117961 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48581081b0 con 0x7f4858060280 2024-02-19T22:15:08.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.282+0000 7f484dffb700 1 -- 172.21.15.184:0/1442117961 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f484801d980 con 0x7f4858060280 2024-02-19T22:15:08.284 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:08.284 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:08.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.284+0000 7f485f53a700 1 -- 172.21.15.184:0/1442117961 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f484402ff10 msgr2=0x7f48440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:08.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.284+0000 7f485f53a700 1 --2- 172.21.15.184:0/1442117961 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f484402ff10 0x7f48440323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f48540096a0 tx=0x7f4854006b40).stop 2024-02-19T22:15:08.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.284+0000 7f485f53a700 1 -- 172.21.15.184:0/1442117961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4858060280 msgr2=0x7f4858106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:08.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.284+0000 7f485f53a700 1 --2- 172.21.15.184:0/1442117961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4858060280 0x7f4858106780 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f48480089f0 tx=0x7f484800b040).stop 2024-02-19T22:15:08.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.284+0000 7f485f53a700 1 -- 172.21.15.184:0/1442117961 shutdown_connections 2024-02-19T22:15:08.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.284+0000 7f485f53a700 1 --2- 172.21.15.184:0/1442117961 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f484402ff10 0x7f48440323d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:08.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.284+0000 7f485f53a700 1 --2- 172.21.15.184:0/1442117961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4858060280 0x7f4858106780 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:08.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.284+0000 7f485f53a700 1 -- 172.21.15.184:0/1442117961 >> 172.21.15.184:0/1442117961 conn(0x7f48580f0b70 msgr2=0x7f48580f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:08.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.284+0000 7f485f53a700 1 -- 172.21.15.184:0/1442117961 shutdown_connections 2024-02-19T22:15:08.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:08.284+0000 7f485f53a700 1 -- 172.21.15.184:0/1442117961 wait complete. 2024-02-19T22:15:08.287 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:09.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:08 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:08.283709+0000 mon.smithi176 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.184:0/1442117961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:10.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:10.042 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:10.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:07.889654+0000 mgr.smithi176.faoogw (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:11.951 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.949+0000 7f11affff700 1 -- 172.21.15.184:0/2558055709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b00601e0 msgr2=0x7f11b00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:11.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.949+0000 7f11affff700 1 --2- 172.21.15.184:0/2558055709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b00601e0 0x7f11b00f8f20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f11a4009fb0 tx=0x7f11a4009b30).stop 2024-02-19T22:15:11.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.949+0000 7f11affff700 1 -- 172.21.15.184:0/2558055709 shutdown_connections 2024-02-19T22:15:11.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.949+0000 7f11affff700 1 --2- 172.21.15.184:0/2558055709 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b00601e0 0x7f11b00f8f20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:11.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.949+0000 7f11affff700 1 -- 172.21.15.184:0/2558055709 >> 172.21.15.184:0/2558055709 conn(0x7f11b00f09a0 msgr2=0x7f11b00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:11.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.949+0000 7f11affff700 1 -- 172.21.15.184:0/2558055709 shutdown_connections 2024-02-19T22:15:11.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.949+0000 7f11affff700 1 -- 172.21.15.184:0/2558055709 wait complete. 2024-02-19T22:15:11.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.950+0000 7f11affff700 1 Processor -- start 2024-02-19T22:15:11.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.950+0000 7f11affff700 1 -- start start 2024-02-19T22:15:11.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.950+0000 7f11affff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b00601e0 0x7f11b0106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:11.952 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.950+0000 7f11affff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b0106af0 con 0x7f11b00601e0 2024-02-19T22:15:11.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.950+0000 7f11aeffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b00601e0 0x7f11b0106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:11.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.950+0000 7f11aeffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b00601e0 0x7f11b0106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:32936/0 (socket says 172.21.15.184:32936) 2024-02-19T22:15:11.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.950+0000 7f11aeffd700 1 -- 172.21.15.184:0/4037549762 learned_addr learned my addr 172.21.15.184:0/4037549762 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:11.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.950+0000 7f11aeffd700 1 -- 172.21.15.184:0/4037549762 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a4005040 con 0x7f11b00601e0 2024-02-19T22:15:11.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.951+0000 7f11aeffd700 1 --2- 172.21.15.184:0/4037549762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b00601e0 0x7f11b0106600 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f11a4005010 tx=0x7f11a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:11.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.951+0000 7f11977fe700 1 -- 172.21.15.184:0/4037549762 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11a4009320 con 0x7f11b00601e0 2024-02-19T22:15:11.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.951+0000 7f11affff700 1 -- 172.21.15.184:0/4037549762 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b0106cf0 con 0x7f11b00601e0 2024-02-19T22:15:11.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.951+0000 7f11977fe700 1 -- 172.21.15.184:0/4037549762 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11a4013380 con 0x7f11b00601e0 2024-02-19T22:15:11.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.951+0000 7f11977fe700 1 -- 172.21.15.184:0/4037549762 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11a4014e50 con 0x7f11b00601e0 2024-02-19T22:15:11.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.951+0000 7f11affff700 1 -- 172.21.15.184:0/4037549762 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b0107180 con 0x7f11b00601e0 2024-02-19T22:15:11.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.952+0000 7f11977fe700 1 -- 172.21.15.184:0/4037549762 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11a4014420 con 0x7f11b00601e0 2024-02-19T22:15:11.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.952+0000 7f11affff700 1 -- 172.21.15.184:0/4037549762 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b0045ad0 con 0x7f11b00601e0 2024-02-19T22:15:11.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.952+0000 7f11977fe700 1 --2- 172.21.15.184:0/4037549762 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f119802ff10 0x7f11980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:11.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.952+0000 7f11977fe700 1 -- 172.21.15.184:0/4037549762 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f11a4040f40 con 0x7f11b00601e0 2024-02-19T22:15:11.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.952+0000 7f11ae7fc700 1 --2- 172.21.15.184:0/4037549762 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f119802ff10 0x7f11980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:11.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.952+0000 7f11ae7fc700 1 --2- 172.21.15.184:0/4037549762 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f119802ff10 0x7f11980323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f11a00096a0 tx=0x7f11a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:11.958 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:11.957+0000 7f11977fe700 1 -- 172.21.15.184:0/4037549762 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11a40095f0 con 0x7f11b00601e0 2024-02-19T22:15:12.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:09.890016+0000 mgr.smithi176.faoogw (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:12.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:12.161+0000 7f11affff700 1 -- 172.21.15.184:0/4037549762 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11b0108030 con 0x7f11b00601e0 2024-02-19T22:15:12.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:12.163+0000 7f11977fe700 1 -- 172.21.15.184:0/4037549762 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f11a401db70 con 0x7f11b00601e0 2024-02-19T22:15:12.164 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:12.164 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:12.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:12.163+0000 7f11affff700 1 -- 172.21.15.184:0/4037549762 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f119802ff10 msgr2=0x7f11980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:12.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:12.163+0000 7f11affff700 1 --2- 172.21.15.184:0/4037549762 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f119802ff10 0x7f11980323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f11a00096a0 tx=0x7f11a0006b40).stop 2024-02-19T22:15:12.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:12.163+0000 7f11affff700 1 -- 172.21.15.184:0/4037549762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b00601e0 msgr2=0x7f11b0106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:12.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:12.163+0000 7f11affff700 1 --2- 172.21.15.184:0/4037549762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b00601e0 0x7f11b0106600 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f11a4005010 tx=0x7f11a400b040).stop 2024-02-19T22:15:12.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:12.163+0000 7f11affff700 1 -- 172.21.15.184:0/4037549762 shutdown_connections 2024-02-19T22:15:12.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:12.163+0000 7f11affff700 1 --2- 172.21.15.184:0/4037549762 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f119802ff10 0x7f11980323d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:12.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:12.163+0000 7f11affff700 1 --2- 172.21.15.184:0/4037549762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b00601e0 0x7f11b0106600 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:12.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:12.163+0000 7f11affff700 1 -- 172.21.15.184:0/4037549762 >> 172.21.15.184:0/4037549762 conn(0x7f11b00f09a0 msgr2=0x7f11b00f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:12.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:12.163+0000 7f11affff700 1 -- 172.21.15.184:0/4037549762 shutdown_connections 2024-02-19T22:15:12.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:12.163+0000 7f11affff700 1 -- 172.21.15.184:0/4037549762 wait complete. 2024-02-19T22:15:12.166 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:13.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:12 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:12.163195+0000 mon.smithi176 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.184:0/4037549762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:13.907 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:13.907 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:14.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:13 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:11.890305+0000 mgr.smithi176.faoogw (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:15.839 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.838+0000 7fb6e28cc700 1 -- 172.21.15.184:0/702720935 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6dc0f5410 msgr2=0x7fb6dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:15.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.838+0000 7fb6e28cc700 1 --2- 172.21.15.184:0/702720935 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6dc0f5410 0x7fb6dc0f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fb6cc009fb0 tx=0x7fb6cc009b30).stop 2024-02-19T22:15:15.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.838+0000 7fb6e28cc700 1 -- 172.21.15.184:0/702720935 shutdown_connections 2024-02-19T22:15:15.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.838+0000 7fb6e28cc700 1 --2- 172.21.15.184:0/702720935 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6dc0f5410 0x7fb6dc0f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:15.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.838+0000 7fb6e28cc700 1 -- 172.21.15.184:0/702720935 >> 172.21.15.184:0/702720935 conn(0x7fb6dc0f0b70 msgr2=0x7fb6dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:15.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.838+0000 7fb6e28cc700 1 -- 172.21.15.184:0/702720935 shutdown_connections 2024-02-19T22:15:15.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.838+0000 7fb6e28cc700 1 -- 172.21.15.184:0/702720935 wait complete. 2024-02-19T22:15:15.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.839+0000 7fb6e28cc700 1 Processor -- start 2024-02-19T22:15:15.840 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.839+0000 7fb6e28cc700 1 -- start start 2024-02-19T22:15:15.842 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.839+0000 7fb6e28cc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6dc0f5410 0x7fb6dc1094d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:15.842 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.839+0000 7fb6e28cc700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc060280 con 0x7fb6dc0f5410 2024-02-19T22:15:15.842 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.839+0000 7fb6dbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6dc0f5410 0x7fb6dc1094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:15.842 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.839+0000 7fb6dbfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6dc0f5410 0x7fb6dc1094d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:32962/0 (socket says 172.21.15.184:32962) 2024-02-19T22:15:15.842 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.839+0000 7fb6dbfff700 1 -- 172.21.15.184:0/1606886166 learned_addr learned my addr 172.21.15.184:0/1606886166 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:15.842 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.840+0000 7fb6dbfff700 1 -- 172.21.15.184:0/1606886166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc005040 con 0x7fb6dc0f5410 2024-02-19T22:15:15.842 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.840+0000 7fb6dbfff700 1 --2- 172.21.15.184:0/1606886166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6dc0f5410 0x7fb6dc1094d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fb6cc005010 tx=0x7fb6cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:15.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.840+0000 7fb6d8ff9700 1 -- 172.21.15.184:0/1606886166 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6cc009320 con 0x7fb6dc0f5410 2024-02-19T22:15:15.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.840+0000 7fb6e28cc700 1 -- 172.21.15.184:0/1606886166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc060480 con 0x7fb6dc0f5410 2024-02-19T22:15:15.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.840+0000 7fb6d8ff9700 1 -- 172.21.15.184:0/1606886166 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6cc013380 con 0x7fb6dc0f5410 2024-02-19T22:15:15.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.840+0000 7fb6d8ff9700 1 -- 172.21.15.184:0/1606886166 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6cc014e50 con 0x7fb6dc0f5410 2024-02-19T22:15:15.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.840+0000 7fb6e28cc700 1 -- 172.21.15.184:0/1606886166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc109c00 con 0x7fb6dc0f5410 2024-02-19T22:15:15.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.841+0000 7fb6d8ff9700 1 -- 172.21.15.184:0/1606886166 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6cc0263a0 con 0x7fb6dc0f5410 2024-02-19T22:15:15.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.841+0000 7fb6d8ff9700 1 --2- 172.21.15.184:0/1606886166 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb6c402ff10 0x7fb6c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:15.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.841+0000 7fb6d8ff9700 1 -- 172.21.15.184:0/1606886166 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb6cc040fb0 con 0x7fb6dc0f5410 2024-02-19T22:15:15.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.841+0000 7fb6e28cc700 1 -- 172.21.15.184:0/1606886166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6dc045ad0 con 0x7fb6dc0f5410 2024-02-19T22:15:15.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.841+0000 7fb6db7fe700 1 --2- 172.21.15.184:0/1606886166 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb6c402ff10 0x7fb6c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:15.843 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.842+0000 7fb6db7fe700 1 --2- 172.21.15.184:0/1606886166 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb6c402ff10 0x7fb6c40323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb6d00096a0 tx=0x7fb6d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:15.846 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:15.846+0000 7fb6d8ff9700 1 -- 172.21.15.184:0/1606886166 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6cc012920 con 0x7fb6dc0f5410 2024-02-19T22:15:16.051 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:16.051+0000 7fb6e28cc700 1 -- 172.21.15.184:0/1606886166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6dc10a3a0 con 0x7fb6dc0f5410 2024-02-19T22:15:16.052 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:16.051+0000 7fb6d8ff9700 1 -- 172.21.15.184:0/1606886166 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb6cc012330 con 0x7fb6dc0f5410 2024-02-19T22:15:16.052 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:16.052 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:16.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:16.053+0000 7fb6e28cc700 1 -- 172.21.15.184:0/1606886166 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb6c402ff10 msgr2=0x7fb6c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:16.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:16.053+0000 7fb6e28cc700 1 --2- 172.21.15.184:0/1606886166 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb6c402ff10 0x7fb6c40323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb6d00096a0 tx=0x7fb6d0006b40).stop 2024-02-19T22:15:16.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:16.053+0000 7fb6e28cc700 1 -- 172.21.15.184:0/1606886166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6dc0f5410 msgr2=0x7fb6dc1094d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:16.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:16.053+0000 7fb6e28cc700 1 --2- 172.21.15.184:0/1606886166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6dc0f5410 0x7fb6dc1094d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fb6cc005010 tx=0x7fb6cc00b040).stop 2024-02-19T22:15:16.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:16.053+0000 7fb6e28cc700 1 -- 172.21.15.184:0/1606886166 shutdown_connections 2024-02-19T22:15:16.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:16.053+0000 7fb6e28cc700 1 --2- 172.21.15.184:0/1606886166 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb6c402ff10 0x7fb6c40323d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:16.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:16.053+0000 7fb6e28cc700 1 --2- 172.21.15.184:0/1606886166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6dc0f5410 0x7fb6dc1094d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:16.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:16.053+0000 7fb6e28cc700 1 -- 172.21.15.184:0/1606886166 >> 172.21.15.184:0/1606886166 conn(0x7fb6dc0f0b70 msgr2=0x7fb6dc0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:16.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:16.053+0000 7fb6e28cc700 1 -- 172.21.15.184:0/1606886166 shutdown_connections 2024-02-19T22:15:16.054 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:16.053+0000 7fb6e28cc700 1 -- 172.21.15.184:0/1606886166 wait complete. 2024-02-19T22:15:16.055 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:16.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:13.890663+0000 mgr.smithi176.faoogw (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:17.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:16 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:16.051798+0000 mon.smithi176 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.184:0/1606886166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:17.779 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:17.779 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:18.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:17 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:15.890960+0000 mgr.smithi176.faoogw (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:19.667 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.664+0000 7ff576b70700 1 -- 172.21.15.184:0/3178103154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5700f6510 msgr2=0x7ff5700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:19.667 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.664+0000 7ff576b70700 1 --2- 172.21.15.184:0/3178103154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5700f6510 0x7ff5700f6930 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7ff568009960 tx=0x7ff568004d10).stop 2024-02-19T22:15:19.667 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.664+0000 7ff576b70700 1 -- 172.21.15.184:0/3178103154 shutdown_connections 2024-02-19T22:15:19.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.664+0000 7ff576b70700 1 --2- 172.21.15.184:0/3178103154 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5700f6510 0x7ff5700f6930 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:19.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.664+0000 7ff576b70700 1 -- 172.21.15.184:0/3178103154 >> 172.21.15.184:0/3178103154 conn(0x7ff5700f1c70 msgr2=0x7ff5700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:19.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.664+0000 7ff576b70700 1 -- 172.21.15.184:0/3178103154 shutdown_connections 2024-02-19T22:15:19.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.664+0000 7ff576b70700 1 -- 172.21.15.184:0/3178103154 wait complete. 2024-02-19T22:15:19.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.665+0000 7ff576b70700 1 Processor -- start 2024-02-19T22:15:19.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.665+0000 7ff576b70700 1 -- start start 2024-02-19T22:15:19.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.665+0000 7ff576b70700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5700f6510 0x7ff5701067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:19.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.665+0000 7ff576b70700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff570106cd0 con 0x7ff5700f6510 2024-02-19T22:15:19.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.665+0000 7ff57490c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5700f6510 0x7ff5701067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:19.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.665+0000 7ff57490c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5700f6510 0x7ff5701067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:33470/0 (socket says 172.21.15.184:33470) 2024-02-19T22:15:19.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.665+0000 7ff57490c700 1 -- 172.21.15.184:0/1197299773 learned_addr learned my addr 172.21.15.184:0/1197299773 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:19.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.665+0000 7ff57490c700 1 -- 172.21.15.184:0/1197299773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff568009640 con 0x7ff5700f6510 2024-02-19T22:15:19.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.665+0000 7ff57490c700 1 --2- 172.21.15.184:0/1197299773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5700f6510 0x7ff5701067e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7ff568004b70 tx=0x7ff568004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:19.669 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.666+0000 7ff56d7fa700 1 -- 172.21.15.184:0/1197299773 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff568018070 con 0x7ff5700f6510 2024-02-19T22:15:19.669 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.666+0000 7ff576b70700 1 -- 172.21.15.184:0/1197299773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff570106ed0 con 0x7ff5700f6510 2024-02-19T22:15:19.669 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.666+0000 7ff56d7fa700 1 -- 172.21.15.184:0/1197299773 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff56800b970 con 0x7ff5700f6510 2024-02-19T22:15:19.669 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.666+0000 7ff56d7fa700 1 -- 172.21.15.184:0/1197299773 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff56801c560 con 0x7ff5700f6510 2024-02-19T22:15:19.669 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.666+0000 7ff576b70700 1 -- 172.21.15.184:0/1197299773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff570107360 con 0x7ff5700f6510 2024-02-19T22:15:19.670 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.666+0000 7ff56d7fa700 1 -- 172.21.15.184:0/1197299773 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff56801c6c0 con 0x7ff5700f6510 2024-02-19T22:15:19.670 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.667+0000 7ff56d7fa700 1 --2- 172.21.15.184:0/1197299773 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff55802ff60 0x7ff558032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:19.670 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.667+0000 7ff56d7fa700 1 -- 172.21.15.184:0/1197299773 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff5680412b0 con 0x7ff5700f6510 2024-02-19T22:15:19.670 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.667+0000 7ff576b70700 1 -- 172.21.15.184:0/1197299773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff570045ad0 con 0x7ff5700f6510 2024-02-19T22:15:19.670 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.667+0000 7ff56ffff700 1 --2- 172.21.15.184:0/1197299773 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff55802ff60 0x7ff558032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:19.670 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.667+0000 7ff56ffff700 1 --2- 172.21.15.184:0/1197299773 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff55802ff60 0x7ff558032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff5640096a0 tx=0x7ff564006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:19.673 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.671+0000 7ff56d7fa700 1 -- 172.21.15.184:0/1197299773 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff5680127f0 con 0x7ff5700f6510 2024-02-19T22:15:19.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.877+0000 7ff576b70700 1 -- 172.21.15.184:0/1197299773 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff5701080a0 con 0x7ff5700f6510 2024-02-19T22:15:19.879 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.877+0000 7ff56d7fa700 1 -- 172.21.15.184:0/1197299773 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff568019230 con 0x7ff5700f6510 2024-02-19T22:15:19.879 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:19.879 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:19.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.879+0000 7ff576b70700 1 -- 172.21.15.184:0/1197299773 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff55802ff60 msgr2=0x7ff558032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:19.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.879+0000 7ff576b70700 1 --2- 172.21.15.184:0/1197299773 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff55802ff60 0x7ff558032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff5640096a0 tx=0x7ff564006b40).stop 2024-02-19T22:15:19.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.879+0000 7ff576b70700 1 -- 172.21.15.184:0/1197299773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5700f6510 msgr2=0x7ff5701067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:19.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.879+0000 7ff576b70700 1 --2- 172.21.15.184:0/1197299773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5700f6510 0x7ff5701067e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7ff568004b70 tx=0x7ff568004d10).stop 2024-02-19T22:15:19.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.879+0000 7ff576b70700 1 -- 172.21.15.184:0/1197299773 shutdown_connections 2024-02-19T22:15:19.881 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.879+0000 7ff576b70700 1 --2- 172.21.15.184:0/1197299773 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff55802ff60 0x7ff558032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:19.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.879+0000 7ff576b70700 1 --2- 172.21.15.184:0/1197299773 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff5700f6510 0x7ff5701067e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:19.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.879+0000 7ff576b70700 1 -- 172.21.15.184:0/1197299773 >> 172.21.15.184:0/1197299773 conn(0x7ff5700f1c70 msgr2=0x7ff5700f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:19.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.879+0000 7ff576b70700 1 -- 172.21.15.184:0/1197299773 shutdown_connections 2024-02-19T22:15:19.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:19.879+0000 7ff576b70700 1 -- 172.21.15.184:0/1197299773 wait complete. 2024-02-19T22:15:19.882 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:20.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:17.891298+0000 mgr.smithi176.faoogw (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:21.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:20 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:19.878780+0000 mon.smithi176 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.184:0/1197299773' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:21.656 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:21.656 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:22.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:19.891654+0000 mgr.smithi176.faoogw (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:23.704 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.688+0000 7f10c327a700 1 -- 172.21.15.184:0/2266177293 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10bc0f5410 msgr2=0x7f10bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:23.704 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.688+0000 7f10c327a700 1 --2- 172.21.15.184:0/2266177293 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10bc0f5410 0x7f10bc0f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f10ac009fb0 tx=0x7f10ac009b30).stop 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.688+0000 7f10c327a700 1 -- 172.21.15.184:0/2266177293 shutdown_connections 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.688+0000 7f10c327a700 1 --2- 172.21.15.184:0/2266177293 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10bc0f5410 0x7f10bc0f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.688+0000 7f10c327a700 1 -- 172.21.15.184:0/2266177293 >> 172.21.15.184:0/2266177293 conn(0x7f10bc0f0b70 msgr2=0x7f10bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.688+0000 7f10c327a700 1 -- 172.21.15.184:0/2266177293 shutdown_connections 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.688+0000 7f10c327a700 1 -- 172.21.15.184:0/2266177293 wait complete. 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.689+0000 7f10c327a700 1 Processor -- start 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.689+0000 7f10c327a700 1 -- start start 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.689+0000 7f10c327a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10bc0f5410 0x7f10bc1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.689+0000 7f10c327a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc060240 con 0x7f10bc0f5410 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.689+0000 7f10c1016700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10bc0f5410 0x7f10bc1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.689+0000 7f10c1016700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10bc0f5410 0x7f10bc1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:33486/0 (socket says 172.21.15.184:33486) 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.689+0000 7f10c1016700 1 -- 172.21.15.184:0/782851097 learned_addr learned my addr 172.21.15.184:0/782851097 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.689+0000 7f10c1016700 1 -- 172.21.15.184:0/782851097 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ac005040 con 0x7f10bc0f5410 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.689+0000 7f10c1016700 1 --2- 172.21.15.184:0/782851097 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10bc0f5410 0x7f10bc1094e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f10ac005010 tx=0x7f10ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:23.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.689+0000 7f10b1ffb700 1 -- 172.21.15.184:0/782851097 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10ac009320 con 0x7f10bc0f5410 2024-02-19T22:15:23.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.689+0000 7f10c327a700 1 -- 172.21.15.184:0/782851097 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10bc060440 con 0x7f10bc0f5410 2024-02-19T22:15:23.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.690+0000 7f10b1ffb700 1 -- 172.21.15.184:0/782851097 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10ac013380 con 0x7f10bc0f5410 2024-02-19T22:15:23.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.690+0000 7f10b1ffb700 1 -- 172.21.15.184:0/782851097 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10ac014e50 con 0x7f10bc0f5410 2024-02-19T22:15:23.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.690+0000 7f10c327a700 1 -- 172.21.15.184:0/782851097 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10bc109b90 con 0x7f10bc0f5410 2024-02-19T22:15:23.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.691+0000 7f10b1ffb700 1 -- 172.21.15.184:0/782851097 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10ac0095f0 con 0x7f10bc0f5410 2024-02-19T22:15:23.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.691+0000 7f10c327a700 1 -- 172.21.15.184:0/782851097 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10bc045ad0 con 0x7f10bc0f5410 2024-02-19T22:15:23.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.691+0000 7f10b1ffb700 1 --2- 172.21.15.184:0/782851097 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f10a802ff10 0x7f10a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:23.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.691+0000 7f10b1ffb700 1 -- 172.21.15.184:0/782851097 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f10ac040e70 con 0x7f10bc0f5410 2024-02-19T22:15:23.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.691+0000 7f10c0815700 1 --2- 172.21.15.184:0/782851097 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f10a802ff10 0x7f10a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:23.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.691+0000 7f10c0815700 1 --2- 172.21.15.184:0/782851097 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f10a802ff10 0x7f10a80323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f10b80096a0 tx=0x7f10b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:23.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.695+0000 7f10b1ffb700 1 -- 172.21.15.184:0/782851097 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10ac014420 con 0x7f10bc0f5410 2024-02-19T22:15:23.905 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.903+0000 7f10c327a700 1 -- 172.21.15.184:0/782851097 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10bc10a3b0 con 0x7f10bc0f5410 2024-02-19T22:15:23.906 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.904+0000 7f10b1ffb700 1 -- 172.21.15.184:0/782851097 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f10ac010230 con 0x7f10bc0f5410 2024-02-19T22:15:23.906 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:23.906 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:23.908 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.906+0000 7f10c327a700 1 -- 172.21.15.184:0/782851097 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f10a802ff10 msgr2=0x7f10a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:23.908 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.906+0000 7f10c327a700 1 --2- 172.21.15.184:0/782851097 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f10a802ff10 0x7f10a80323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f10b80096a0 tx=0x7f10b8006b40).stop 2024-02-19T22:15:23.908 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.906+0000 7f10c327a700 1 -- 172.21.15.184:0/782851097 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10bc0f5410 msgr2=0x7f10bc1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:23.908 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.906+0000 7f10c327a700 1 --2- 172.21.15.184:0/782851097 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10bc0f5410 0x7f10bc1094e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f10ac005010 tx=0x7f10ac00b040).stop 2024-02-19T22:15:23.908 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.906+0000 7f10c327a700 1 -- 172.21.15.184:0/782851097 shutdown_connections 2024-02-19T22:15:23.908 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.906+0000 7f10c327a700 1 --2- 172.21.15.184:0/782851097 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f10a802ff10 0x7f10a80323d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:23.908 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.906+0000 7f10c327a700 1 --2- 172.21.15.184:0/782851097 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10bc0f5410 0x7f10bc1094e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:23.908 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.906+0000 7f10c327a700 1 -- 172.21.15.184:0/782851097 >> 172.21.15.184:0/782851097 conn(0x7f10bc0f0b70 msgr2=0x7f10bc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:23.908 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.906+0000 7f10c327a700 1 -- 172.21.15.184:0/782851097 shutdown_connections 2024-02-19T22:15:23.908 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:23.906+0000 7f10c327a700 1 -- 172.21.15.184:0/782851097 wait complete. 2024-02-19T22:15:23.908 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:24.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:21.891932+0000 mgr.smithi176.faoogw (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:25.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:24 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:23.905426+0000 mon.smithi176 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.184:0/782851097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:25.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:25.653 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:26.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:23.892216+0000 mgr.smithi176.faoogw (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:27.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.583+0000 7f0af434d700 1 -- 172.21.15.184:0/2514947111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0aec0f5410 msgr2=0x7f0aec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:27.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.583+0000 7f0af434d700 1 --2- 172.21.15.184:0/2514947111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0aec0f5410 0x7f0aec0f5830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f0ae0009fb0 tx=0x7f0ae0009b30).stop 2024-02-19T22:15:27.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.583+0000 7f0af434d700 1 -- 172.21.15.184:0/2514947111 shutdown_connections 2024-02-19T22:15:27.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.583+0000 7f0af434d700 1 --2- 172.21.15.184:0/2514947111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0aec0f5410 0x7f0aec0f5830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:27.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.583+0000 7f0af434d700 1 -- 172.21.15.184:0/2514947111 >> 172.21.15.184:0/2514947111 conn(0x7f0aec0f0b70 msgr2=0x7f0aec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:27.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.583+0000 7f0af434d700 1 -- 172.21.15.184:0/2514947111 shutdown_connections 2024-02-19T22:15:27.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.583+0000 7f0af434d700 1 -- 172.21.15.184:0/2514947111 wait complete. 2024-02-19T22:15:27.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.584+0000 7f0af434d700 1 Processor -- start 2024-02-19T22:15:27.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.584+0000 7f0af434d700 1 -- start start 2024-02-19T22:15:27.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.584+0000 7f0af434d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0aec0f5410 0x7f0aec0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:27.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.584+0000 7f0af434d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec0fb960 con 0x7f0aec0f5410 2024-02-19T22:15:27.586 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.584+0000 7f0af20e9700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0aec0f5410 0x7f0aec0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:27.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.584+0000 7f0af20e9700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0aec0f5410 0x7f0aec0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:37078/0 (socket says 172.21.15.184:37078) 2024-02-19T22:15:27.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.584+0000 7f0af20e9700 1 -- 172.21.15.184:0/3815097393 learned_addr learned my addr 172.21.15.184:0/3815097393 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:27.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.584+0000 7f0af20e9700 1 -- 172.21.15.184:0/3815097393 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae0005040 con 0x7f0aec0f5410 2024-02-19T22:15:27.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.584+0000 7f0af20e9700 1 --2- 172.21.15.184:0/3815097393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0aec0f5410 0x7f0aec0fd4e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f0ae0005010 tx=0x7f0ae000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:27.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.585+0000 7f0adeffd700 1 -- 172.21.15.184:0/3815097393 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ae0009320 con 0x7f0aec0f5410 2024-02-19T22:15:27.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.585+0000 7f0adeffd700 1 -- 172.21.15.184:0/3815097393 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ae0013380 con 0x7f0aec0f5410 2024-02-19T22:15:27.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.585+0000 7f0af434d700 1 -- 172.21.15.184:0/3815097393 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aec0fbb00 con 0x7f0aec0f5410 2024-02-19T22:15:27.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.585+0000 7f0adeffd700 1 -- 172.21.15.184:0/3815097393 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ae0014e50 con 0x7f0aec0f5410 2024-02-19T22:15:27.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.585+0000 7f0af434d700 1 -- 172.21.15.184:0/3815097393 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aec0fbf90 con 0x7f0aec0f5410 2024-02-19T22:15:27.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.585+0000 7f0adeffd700 1 -- 172.21.15.184:0/3815097393 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ae00263a0 con 0x7f0aec0f5410 2024-02-19T22:15:27.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.586+0000 7f0adeffd700 1 --2- 172.21.15.184:0/3815097393 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f0ad802ff10 0x7f0ad80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:27.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.586+0000 7f0adeffd700 1 -- 172.21.15.184:0/3815097393 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ae0040e90 con 0x7f0aec0f5410 2024-02-19T22:15:27.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.586+0000 7f0af434d700 1 -- 172.21.15.184:0/3815097393 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aec059100 con 0x7f0aec0f5410 2024-02-19T22:15:27.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.586+0000 7f0af18e8700 1 --2- 172.21.15.184:0/3815097393 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f0ad802ff10 0x7f0ad80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:27.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.586+0000 7f0af18e8700 1 --2- 172.21.15.184:0/3815097393 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f0ad802ff10 0x7f0ad80323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f0ae80096a0 tx=0x7f0ae8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:27.592 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.590+0000 7f0adeffd700 1 -- 172.21.15.184:0/3815097393 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ae0012920 con 0x7f0aec0f5410 2024-02-19T22:15:27.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.797+0000 7f0af434d700 1 -- 172.21.15.184:0/3815097393 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0aec045ad0 con 0x7f0aec0f5410 2024-02-19T22:15:27.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.798+0000 7f0adeffd700 1 -- 172.21.15.184:0/3815097393 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0ae0012330 con 0x7f0aec0f5410 2024-02-19T22:15:27.800 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:27.801 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:27.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.800+0000 7f0af434d700 1 -- 172.21.15.184:0/3815097393 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f0ad802ff10 msgr2=0x7f0ad80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:27.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.800+0000 7f0af434d700 1 --2- 172.21.15.184:0/3815097393 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f0ad802ff10 0x7f0ad80323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f0ae80096a0 tx=0x7f0ae8006b40).stop 2024-02-19T22:15:27.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.800+0000 7f0af434d700 1 -- 172.21.15.184:0/3815097393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0aec0f5410 msgr2=0x7f0aec0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:27.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.800+0000 7f0af434d700 1 --2- 172.21.15.184:0/3815097393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0aec0f5410 0x7f0aec0fd4e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f0ae0005010 tx=0x7f0ae000b040).stop 2024-02-19T22:15:27.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.800+0000 7f0af434d700 1 -- 172.21.15.184:0/3815097393 shutdown_connections 2024-02-19T22:15:27.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.800+0000 7f0af434d700 1 --2- 172.21.15.184:0/3815097393 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f0ad802ff10 0x7f0ad80323d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:27.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.800+0000 7f0af434d700 1 --2- 172.21.15.184:0/3815097393 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0aec0f5410 0x7f0aec0fd4e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:27.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.800+0000 7f0af434d700 1 -- 172.21.15.184:0/3815097393 >> 172.21.15.184:0/3815097393 conn(0x7f0aec0f0b70 msgr2=0x7f0aec05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:27.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.800+0000 7f0af434d700 1 -- 172.21.15.184:0/3815097393 shutdown_connections 2024-02-19T22:15:27.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:27.800+0000 7f0af434d700 1 -- 172.21.15.184:0/3815097393 wait complete. 2024-02-19T22:15:27.803 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:28.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:25.892598+0000 mgr.smithi176.faoogw (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:28.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:27.799626+0000 mon.smithi176 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.184:0/3815097393' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:29.526 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:29.526 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:30.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:29 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:27.892979+0000 mgr.smithi176.faoogw (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:31.450 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.447+0000 7f8efc649700 1 -- 172.21.15.184:0/1529803031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ef40f4740 msgr2=0x7f8ef40f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:31.450 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.447+0000 7f8efc649700 1 --2- 172.21.15.184:0/1529803031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ef40f4740 0x7f8ef40f4b60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f8ee4009fb0 tx=0x7f8ee4009b30).stop 2024-02-19T22:15:31.450 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.447+0000 7f8efc649700 1 -- 172.21.15.184:0/1529803031 shutdown_connections 2024-02-19T22:15:31.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.447+0000 7f8efc649700 1 --2- 172.21.15.184:0/1529803031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ef40f4740 0x7f8ef40f4b60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:31.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.447+0000 7f8efc649700 1 -- 172.21.15.184:0/1529803031 >> 172.21.15.184:0/1529803031 conn(0x7f8ef40f0260 msgr2=0x7f8ef40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:31.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.447+0000 7f8efc649700 1 -- 172.21.15.184:0/1529803031 shutdown_connections 2024-02-19T22:15:31.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.447+0000 7f8efc649700 1 -- 172.21.15.184:0/1529803031 wait complete. 2024-02-19T22:15:31.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.447+0000 7f8efc649700 1 Processor -- start 2024-02-19T22:15:31.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.447+0000 7f8efc649700 1 -- start start 2024-02-19T22:15:31.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.447+0000 7f8efc649700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ef40f4740 0x7f8ef41089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:31.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.448+0000 7f8efc649700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef4108ed0 con 0x7f8ef40f4740 2024-02-19T22:15:31.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.448+0000 7f8efa3e5700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ef40f4740 0x7f8ef41089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:31.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.448+0000 7f8efa3e5700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ef40f4740 0x7f8ef41089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:37090/0 (socket says 172.21.15.184:37090) 2024-02-19T22:15:31.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.448+0000 7f8efa3e5700 1 -- 172.21.15.184:0/2959421634 learned_addr learned my addr 172.21.15.184:0/2959421634 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:31.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.448+0000 7f8efa3e5700 1 -- 172.21.15.184:0/2959421634 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee4005040 con 0x7f8ef40f4740 2024-02-19T22:15:31.451 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.448+0000 7f8efa3e5700 1 --2- 172.21.15.184:0/2959421634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ef40f4740 0x7f8ef41089e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f8ee40089f0 tx=0x7f8ee400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:31.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.448+0000 7f8eeaffd700 1 -- 172.21.15.184:0/2959421634 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ee4009320 con 0x7f8ef40f4740 2024-02-19T22:15:31.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.448+0000 7f8efc649700 1 -- 172.21.15.184:0/2959421634 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef41090d0 con 0x7f8ef40f4740 2024-02-19T22:15:31.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.448+0000 7f8eeaffd700 1 -- 172.21.15.184:0/2959421634 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ee4013380 con 0x7f8ef40f4740 2024-02-19T22:15:31.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.448+0000 7f8eeaffd700 1 -- 172.21.15.184:0/2959421634 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ee401ed80 con 0x7f8ef40f4740 2024-02-19T22:15:31.452 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.448+0000 7f8efc649700 1 -- 172.21.15.184:0/2959421634 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef4109560 con 0x7f8ef40f4740 2024-02-19T22:15:31.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.449+0000 7f8eeaffd700 1 -- 172.21.15.184:0/2959421634 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ee4014420 con 0x7f8ef40f4740 2024-02-19T22:15:31.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.449+0000 7f8efc649700 1 -- 172.21.15.184:0/2959421634 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef4045ad0 con 0x7f8ef40f4740 2024-02-19T22:15:31.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.449+0000 7f8eeaffd700 1 --2- 172.21.15.184:0/2959421634 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8ee002ff60 0x7f8ee0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:31.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.450+0000 7f8eeaffd700 1 -- 172.21.15.184:0/2959421634 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ee4041140 con 0x7f8ef40f4740 2024-02-19T22:15:31.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.450+0000 7f8ef9be4700 1 --2- 172.21.15.184:0/2959421634 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8ee002ff60 0x7f8ee0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:31.453 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.450+0000 7f8ef9be4700 1 --2- 172.21.15.184:0/2959421634 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8ee002ff60 0x7f8ee0032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f8ef00096a0 tx=0x7f8ef0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:31.456 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.454+0000 7f8eeaffd700 1 -- 172.21.15.184:0/2959421634 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ee400fb30 con 0x7f8ef40f4740 2024-02-19T22:15:31.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.660+0000 7f8efc649700 1 -- 172.21.15.184:0/2959421634 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ef40f6c40 con 0x7f8ef40f4740 2024-02-19T22:15:31.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.661+0000 7f8eeaffd700 1 -- 172.21.15.184:0/2959421634 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8ee400fb30 con 0x7f8ef40f4740 2024-02-19T22:15:31.663 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:31.663 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:31.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.663+0000 7f8efc649700 1 -- 172.21.15.184:0/2959421634 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8ee002ff60 msgr2=0x7f8ee0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:31.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.663+0000 7f8efc649700 1 --2- 172.21.15.184:0/2959421634 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8ee002ff60 0x7f8ee0032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f8ef00096a0 tx=0x7f8ef0006b40).stop 2024-02-19T22:15:31.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.663+0000 7f8efc649700 1 -- 172.21.15.184:0/2959421634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ef40f4740 msgr2=0x7f8ef41089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:31.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.663+0000 7f8efc649700 1 --2- 172.21.15.184:0/2959421634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ef40f4740 0x7f8ef41089e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f8ee40089f0 tx=0x7f8ee400b040).stop 2024-02-19T22:15:31.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.663+0000 7f8efc649700 1 -- 172.21.15.184:0/2959421634 shutdown_connections 2024-02-19T22:15:31.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.663+0000 7f8efc649700 1 --2- 172.21.15.184:0/2959421634 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8ee002ff60 0x7f8ee0032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:31.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.663+0000 7f8efc649700 1 --2- 172.21.15.184:0/2959421634 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8ef40f4740 0x7f8ef41089e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:31.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.663+0000 7f8efc649700 1 -- 172.21.15.184:0/2959421634 >> 172.21.15.184:0/2959421634 conn(0x7f8ef40f0260 msgr2=0x7f8ef40f25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:31.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.663+0000 7f8efc649700 1 -- 172.21.15.184:0/2959421634 shutdown_connections 2024-02-19T22:15:31.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:31.663+0000 7f8efc649700 1 -- 172.21.15.184:0/2959421634 wait complete. 2024-02-19T22:15:31.665 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:32.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:29.893330+0000 mgr.smithi176.faoogw (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:32.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:31.662579+0000 mon.smithi176 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.184:0/2959421634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:33.492 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:33.493 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:34.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:31.893679+0000 mgr.smithi176.faoogw (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:35.466 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.463+0000 7f09bba40700 1 -- 172.21.15.184:0/3829997979 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09b40f5410 msgr2=0x7f09b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:35.466 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.463+0000 7f09bba40700 1 --2- 172.21.15.184:0/3829997979 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09b40f5410 0x7f09b40f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f09b0009fb0 tx=0x7f09b0009b30).stop 2024-02-19T22:15:35.466 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.463+0000 7f09bba40700 1 -- 172.21.15.184:0/3829997979 shutdown_connections 2024-02-19T22:15:35.466 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.463+0000 7f09bba40700 1 --2- 172.21.15.184:0/3829997979 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09b40f5410 0x7f09b40f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:35.466 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.463+0000 7f09bba40700 1 -- 172.21.15.184:0/3829997979 >> 172.21.15.184:0/3829997979 conn(0x7f09b40f0b70 msgr2=0x7f09b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:35.466 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.463+0000 7f09bba40700 1 -- 172.21.15.184:0/3829997979 shutdown_connections 2024-02-19T22:15:35.466 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.463+0000 7f09bba40700 1 -- 172.21.15.184:0/3829997979 wait complete. 2024-02-19T22:15:35.466 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.464+0000 7f09bba40700 1 Processor -- start 2024-02-19T22:15:35.466 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.464+0000 7f09bba40700 1 -- start start 2024-02-19T22:15:35.466 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.464+0000 7f09bba40700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09b40f5410 0x7f09b41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:35.467 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.464+0000 7f09bba40700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b4105bd0 con 0x7f09b40f5410 2024-02-19T22:15:35.467 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.465+0000 7f09b97dc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09b40f5410 0x7f09b41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:35.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.465+0000 7f09b97dc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09b40f5410 0x7f09b41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:37116/0 (socket says 172.21.15.184:37116) 2024-02-19T22:15:35.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.465+0000 7f09b97dc700 1 -- 172.21.15.184:0/3949084890 learned_addr learned my addr 172.21.15.184:0/3949084890 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:35.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.465+0000 7f09b97dc700 1 -- 172.21.15.184:0/3949084890 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b0005040 con 0x7f09b40f5410 2024-02-19T22:15:35.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.465+0000 7f09b97dc700 1 --2- 172.21.15.184:0/3949084890 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09b40f5410 0x7f09b41056e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f09b0005010 tx=0x7f09b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:35.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.465+0000 7f09a9ffb700 1 -- 172.21.15.184:0/3949084890 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09b0009320 con 0x7f09b40f5410 2024-02-19T22:15:35.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.465+0000 7f09a9ffb700 1 -- 172.21.15.184:0/3949084890 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09b0013380 con 0x7f09b40f5410 2024-02-19T22:15:35.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.465+0000 7f09bba40700 1 -- 172.21.15.184:0/3949084890 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b4105dd0 con 0x7f09b40f5410 2024-02-19T22:15:35.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.465+0000 7f09a9ffb700 1 -- 172.21.15.184:0/3949084890 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09b0014e50 con 0x7f09b40f5410 2024-02-19T22:15:35.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.466+0000 7f09bba40700 1 -- 172.21.15.184:0/3949084890 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b4106260 con 0x7f09b40f5410 2024-02-19T22:15:35.469 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.466+0000 7f09a9ffb700 1 -- 172.21.15.184:0/3949084890 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09b0014420 con 0x7f09b40f5410 2024-02-19T22:15:35.469 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.466+0000 7f09bba40700 1 -- 172.21.15.184:0/3949084890 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b4045ad0 con 0x7f09b40f5410 2024-02-19T22:15:35.469 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.466+0000 7f09a9ffb700 1 --2- 172.21.15.184:0/3949084890 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f09a002ff10 0x7f09a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:35.469 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.467+0000 7f09a9ffb700 1 -- 172.21.15.184:0/3949084890 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09b0040f70 con 0x7f09b40f5410 2024-02-19T22:15:35.469 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.467+0000 7f09b8fdb700 1 --2- 172.21.15.184:0/3949084890 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f09a002ff10 0x7f09a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:35.469 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.467+0000 7f09b8fdb700 1 --2- 172.21.15.184:0/3949084890 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f09a002ff10 0x7f09a00323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f09a40096a0 tx=0x7f09a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:35.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.471+0000 7f09a9ffb700 1 -- 172.21.15.184:0/3949084890 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09b00095f0 con 0x7f09b40f5410 2024-02-19T22:15:35.680 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.678+0000 7f09bba40700 1 -- 172.21.15.184:0/3949084890 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09b4107110 con 0x7f09b40f5410 2024-02-19T22:15:35.680 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.679+0000 7f09a9ffb700 1 -- 172.21.15.184:0/3949084890 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f09b001db70 con 0x7f09b40f5410 2024-02-19T22:15:35.681 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:35.681 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:35.682 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.680+0000 7f09bba40700 1 -- 172.21.15.184:0/3949084890 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f09a002ff10 msgr2=0x7f09a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:35.682 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.680+0000 7f09bba40700 1 --2- 172.21.15.184:0/3949084890 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f09a002ff10 0x7f09a00323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f09a40096a0 tx=0x7f09a4006b40).stop 2024-02-19T22:15:35.682 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.680+0000 7f09bba40700 1 -- 172.21.15.184:0/3949084890 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09b40f5410 msgr2=0x7f09b41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:35.682 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.680+0000 7f09bba40700 1 --2- 172.21.15.184:0/3949084890 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09b40f5410 0x7f09b41056e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f09b0005010 tx=0x7f09b000b040).stop 2024-02-19T22:15:35.683 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.681+0000 7f09bba40700 1 -- 172.21.15.184:0/3949084890 shutdown_connections 2024-02-19T22:15:35.683 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.681+0000 7f09bba40700 1 --2- 172.21.15.184:0/3949084890 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f09a002ff10 0x7f09a00323d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:35.683 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.681+0000 7f09bba40700 1 --2- 172.21.15.184:0/3949084890 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09b40f5410 0x7f09b41056e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:35.683 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.681+0000 7f09bba40700 1 -- 172.21.15.184:0/3949084890 >> 172.21.15.184:0/3949084890 conn(0x7f09b40f0b70 msgr2=0x7f09b40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:35.683 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.681+0000 7f09bba40700 1 -- 172.21.15.184:0/3949084890 shutdown_connections 2024-02-19T22:15:35.683 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:35.681+0000 7f09bba40700 1 -- 172.21.15.184:0/3949084890 wait complete. 2024-02-19T22:15:35.683 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:36.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:33.893943+0000 mgr.smithi176.faoogw (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:36.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:35.680363+0000 mon.smithi176 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.184:0/3949084890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:35.972706+0000 mon.smithi176 (mon.0) 241 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:15:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:35.974838+0000 mon.smithi176 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:15:37.393 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:37.393 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:38.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:35.894286+0000 mgr.smithi176.faoogw (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:39.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.227+0000 7f18b11be700 1 -- 172.21.15.184:0/570853397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ac0f6510 msgr2=0x7f18ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:39.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.227+0000 7f18b11be700 1 --2- 172.21.15.184:0/570853397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ac0f6510 0x7f18ac0f6930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f18a0009fb0 tx=0x7f18a0009b30).stop 2024-02-19T22:15:39.229 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.227+0000 7f18b11be700 1 -- 172.21.15.184:0/570853397 shutdown_connections 2024-02-19T22:15:39.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.227+0000 7f18b11be700 1 --2- 172.21.15.184:0/570853397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ac0f6510 0x7f18ac0f6930 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:39.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.227+0000 7f18b11be700 1 -- 172.21.15.184:0/570853397 >> 172.21.15.184:0/570853397 conn(0x7f18ac0f1c70 msgr2=0x7f18ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:39.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.227+0000 7f18b11be700 1 -- 172.21.15.184:0/570853397 shutdown_connections 2024-02-19T22:15:39.230 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.227+0000 7f18b11be700 1 -- 172.21.15.184:0/570853397 wait complete. 2024-02-19T22:15:39.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.228+0000 7f18b11be700 1 Processor -- start 2024-02-19T22:15:39.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.228+0000 7f18b11be700 1 -- start start 2024-02-19T22:15:39.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.228+0000 7f18b11be700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ac0f6510 0x7f18ac1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:39.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.228+0000 7f18b11be700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac106cd0 con 0x7f18ac0f6510 2024-02-19T22:15:39.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.229+0000 7f18aad9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ac0f6510 0x7f18ac1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:39.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.229+0000 7f18aad9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ac0f6510 0x7f18ac1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:33098/0 (socket says 172.21.15.184:33098) 2024-02-19T22:15:39.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.229+0000 7f18aad9d700 1 -- 172.21.15.184:0/356195204 learned_addr learned my addr 172.21.15.184:0/356195204 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:39.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.229+0000 7f18aad9d700 1 -- 172.21.15.184:0/356195204 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a0005040 con 0x7f18ac0f6510 2024-02-19T22:15:39.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.229+0000 7f18aad9d700 1 --2- 172.21.15.184:0/356195204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ac0f6510 0x7f18ac1067e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f18a0005010 tx=0x7f18a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:39.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.229+0000 7f18937fe700 1 -- 172.21.15.184:0/356195204 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18a0009320 con 0x7f18ac0f6510 2024-02-19T22:15:39.232 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.229+0000 7f18b11be700 1 -- 172.21.15.184:0/356195204 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac106ed0 con 0x7f18ac0f6510 2024-02-19T22:15:39.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.229+0000 7f18937fe700 1 -- 172.21.15.184:0/356195204 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18a0013380 con 0x7f18ac0f6510 2024-02-19T22:15:39.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.229+0000 7f18937fe700 1 -- 172.21.15.184:0/356195204 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18a0014e50 con 0x7f18ac0f6510 2024-02-19T22:15:39.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.229+0000 7f18b11be700 1 -- 172.21.15.184:0/356195204 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ac107360 con 0x7f18ac0f6510 2024-02-19T22:15:39.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.230+0000 7f18937fe700 1 -- 172.21.15.184:0/356195204 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18a00095f0 con 0x7f18ac0f6510 2024-02-19T22:15:39.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.230+0000 7f18b11be700 1 -- 172.21.15.184:0/356195204 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ac045ad0 con 0x7f18ac0f6510 2024-02-19T22:15:39.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.230+0000 7f18937fe700 1 --2- 172.21.15.184:0/356195204 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f189402ff10 0x7f18940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:39.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.230+0000 7f18937fe700 1 -- 172.21.15.184:0/356195204 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18a0040f80 con 0x7f18ac0f6510 2024-02-19T22:15:39.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.230+0000 7f18aa59c700 1 --2- 172.21.15.184:0/356195204 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f189402ff10 0x7f18940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:39.233 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.231+0000 7f18aa59c700 1 --2- 172.21.15.184:0/356195204 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f189402ff10 0x7f18940323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f189c0096a0 tx=0x7f189c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:39.236 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.235+0000 7f18937fe700 1 -- 172.21.15.184:0/356195204 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18a0014420 con 0x7f18ac0f6510 2024-02-19T22:15:39.441 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.439+0000 7f18b11be700 1 -- 172.21.15.184:0/356195204 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18ac108210 con 0x7f18ac0f6510 2024-02-19T22:15:39.441 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.440+0000 7f18937fe700 1 -- 172.21.15.184:0/356195204 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f18a0010230 con 0x7f18ac0f6510 2024-02-19T22:15:39.442 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:39.442 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:39.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.441+0000 7f18b11be700 1 -- 172.21.15.184:0/356195204 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f189402ff10 msgr2=0x7f18940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:39.443 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.441+0000 7f18b11be700 1 --2- 172.21.15.184:0/356195204 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f189402ff10 0x7f18940323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f189c0096a0 tx=0x7f189c006b40).stop 2024-02-19T22:15:39.444 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.441+0000 7f18b11be700 1 -- 172.21.15.184:0/356195204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ac0f6510 msgr2=0x7f18ac1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:39.444 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.441+0000 7f18b11be700 1 --2- 172.21.15.184:0/356195204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ac0f6510 0x7f18ac1067e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f18a0005010 tx=0x7f18a000b040).stop 2024-02-19T22:15:39.444 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.442+0000 7f18b11be700 1 -- 172.21.15.184:0/356195204 shutdown_connections 2024-02-19T22:15:39.444 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.442+0000 7f18b11be700 1 --2- 172.21.15.184:0/356195204 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f189402ff10 0x7f18940323d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:39.444 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.442+0000 7f18b11be700 1 --2- 172.21.15.184:0/356195204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18ac0f6510 0x7f18ac1067e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:39.444 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.442+0000 7f18b11be700 1 -- 172.21.15.184:0/356195204 >> 172.21.15.184:0/356195204 conn(0x7f18ac0f1c70 msgr2=0x7f18ac0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:39.444 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.442+0000 7f18b11be700 1 -- 172.21.15.184:0/356195204 shutdown_connections 2024-02-19T22:15:39.444 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:39.442+0000 7f18b11be700 1 -- 172.21.15.184:0/356195204 wait complete. 2024-02-19T22:15:39.445 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:40.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:37.894602+0000 mgr.smithi176.faoogw (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:40.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:39.441171+0000 mon.smithi176 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.184:0/356195204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:41.030 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:41.031 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:42.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:39.894924+0000 mgr.smithi176.faoogw (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:42.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.942+0000 7f3d52f65700 1 -- 172.21.15.184:0/717306257 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d4c060220 msgr2=0x7f3d4c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:42.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.942+0000 7f3d52f65700 1 --2- 172.21.15.184:0/717306257 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d4c060220 0x7f3d4c0f9150 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f3d40009fb0 tx=0x7f3d40009b30).stop 2024-02-19T22:15:42.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.942+0000 7f3d52f65700 1 -- 172.21.15.184:0/717306257 shutdown_connections 2024-02-19T22:15:42.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.942+0000 7f3d52f65700 1 --2- 172.21.15.184:0/717306257 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d4c060220 0x7f3d4c0f9150 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:42.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.942+0000 7f3d52f65700 1 -- 172.21.15.184:0/717306257 >> 172.21.15.184:0/717306257 conn(0x7f3d4c0f0b70 msgr2=0x7f3d4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:42.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.942+0000 7f3d52f65700 1 -- 172.21.15.184:0/717306257 shutdown_connections 2024-02-19T22:15:42.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.942+0000 7f3d52f65700 1 -- 172.21.15.184:0/717306257 wait complete. 2024-02-19T22:15:42.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.943+0000 7f3d52f65700 1 Processor -- start 2024-02-19T22:15:42.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.943+0000 7f3d52f65700 1 -- start start 2024-02-19T22:15:42.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.943+0000 7f3d52f65700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d4c060220 0x7f3d4c1067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:42.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.943+0000 7f3d52f65700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c106c90 con 0x7f3d4c060220 2024-02-19T22:15:42.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.944+0000 7f3d50d01700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d4c060220 0x7f3d4c1067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:42.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.944+0000 7f3d50d01700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d4c060220 0x7f3d4c1067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:33116/0 (socket says 172.21.15.184:33116) 2024-02-19T22:15:42.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.944+0000 7f3d50d01700 1 -- 172.21.15.184:0/1835075887 learned_addr learned my addr 172.21.15.184:0/1835075887 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:42.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.944+0000 7f3d50d01700 1 -- 172.21.15.184:0/1835075887 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d40005040 con 0x7f3d4c060220 2024-02-19T22:15:42.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.944+0000 7f3d50d01700 1 --2- 172.21.15.184:0/1835075887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d4c060220 0x7f3d4c1067a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f3d400089f0 tx=0x7f3d4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:42.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.944+0000 7f3d497fa700 1 -- 172.21.15.184:0/1835075887 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d40009320 con 0x7f3d4c060220 2024-02-19T22:15:42.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.944+0000 7f3d497fa700 1 -- 172.21.15.184:0/1835075887 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d40013380 con 0x7f3d4c060220 2024-02-19T22:15:42.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.944+0000 7f3d52f65700 1 -- 172.21.15.184:0/1835075887 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d4c106e90 con 0x7f3d4c060220 2024-02-19T22:15:42.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.944+0000 7f3d497fa700 1 -- 172.21.15.184:0/1835075887 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d40014e20 con 0x7f3d4c060220 2024-02-19T22:15:42.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.944+0000 7f3d52f65700 1 -- 172.21.15.184:0/1835075887 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4c107320 con 0x7f3d4c060220 2024-02-19T22:15:42.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.945+0000 7f3d497fa700 1 -- 172.21.15.184:0/1835075887 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d400095f0 con 0x7f3d4c060220 2024-02-19T22:15:42.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.945+0000 7f3d52f65700 1 -- 172.21.15.184:0/1835075887 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4c045ad0 con 0x7f3d4c060220 2024-02-19T22:15:42.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.945+0000 7f3d497fa700 1 --2- 172.21.15.184:0/1835075887 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3d3c02ff10 0x7f3d3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:42.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.945+0000 7f3d497fa700 1 -- 172.21.15.184:0/1835075887 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d40040f50 con 0x7f3d4c060220 2024-02-19T22:15:42.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.945+0000 7f3d4bfff700 1 --2- 172.21.15.184:0/1835075887 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3d3c02ff10 0x7f3d3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:42.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.946+0000 7f3d4bfff700 1 --2- 172.21.15.184:0/1835075887 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3d3c02ff10 0x7f3d3c0323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f3d380096a0 tx=0x7f3d38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:42.951 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:42.950+0000 7f3d497fa700 1 -- 172.21.15.184:0/1835075887 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d40014420 con 0x7f3d4c060220 2024-02-19T22:15:43.158 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:43.156+0000 7f3d52f65700 1 -- 172.21.15.184:0/1835075887 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d4c1081d0 con 0x7f3d4c060220 2024-02-19T22:15:43.159 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:43.157+0000 7f3d497fa700 1 -- 172.21.15.184:0/1835075887 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3d4001d980 con 0x7f3d4c060220 2024-02-19T22:15:43.159 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:43.159 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:43.161 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:43.159+0000 7f3d52f65700 1 -- 172.21.15.184:0/1835075887 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3d3c02ff10 msgr2=0x7f3d3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:43.161 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:43.159+0000 7f3d52f65700 1 --2- 172.21.15.184:0/1835075887 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3d3c02ff10 0x7f3d3c0323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f3d380096a0 tx=0x7f3d38006b40).stop 2024-02-19T22:15:43.161 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:43.159+0000 7f3d52f65700 1 -- 172.21.15.184:0/1835075887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d4c060220 msgr2=0x7f3d4c1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:43.161 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:43.159+0000 7f3d52f65700 1 --2- 172.21.15.184:0/1835075887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d4c060220 0x7f3d4c1067a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f3d400089f0 tx=0x7f3d4000b040).stop 2024-02-19T22:15:43.161 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:43.159+0000 7f3d52f65700 1 -- 172.21.15.184:0/1835075887 shutdown_connections 2024-02-19T22:15:43.161 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:43.159+0000 7f3d52f65700 1 --2- 172.21.15.184:0/1835075887 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3d3c02ff10 0x7f3d3c0323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:43.161 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:43.159+0000 7f3d52f65700 1 --2- 172.21.15.184:0/1835075887 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d4c060220 0x7f3d4c1067a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:43.161 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:43.159+0000 7f3d52f65700 1 -- 172.21.15.184:0/1835075887 >> 172.21.15.184:0/1835075887 conn(0x7f3d4c0f0b70 msgr2=0x7f3d4c0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:43.161 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:43.159+0000 7f3d52f65700 1 -- 172.21.15.184:0/1835075887 shutdown_connections 2024-02-19T22:15:43.162 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:43.159+0000 7f3d52f65700 1 -- 172.21.15.184:0/1835075887 wait complete. 2024-02-19T22:15:43.162 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:44.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:41.895288+0000 mgr.smithi176.faoogw (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:44.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:43.158547+0000 mon.smithi176 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.184:0/1835075887' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:44.862 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:44.862 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:46.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:43.895674+0000 mgr.smithi176.faoogw (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:46.751 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.748+0000 7f4128f57700 1 -- 172.21.15.184:0/3541580018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41240f5410 msgr2=0x7f41240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:46.751 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.748+0000 7f4128f57700 1 --2- 172.21.15.184:0/3541580018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41240f5410 0x7f41240f5830 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f4114009fb0 tx=0x7f4114009b30).stop 2024-02-19T22:15:46.751 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.748+0000 7f4128f57700 1 -- 172.21.15.184:0/3541580018 shutdown_connections 2024-02-19T22:15:46.751 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.748+0000 7f4128f57700 1 --2- 172.21.15.184:0/3541580018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41240f5410 0x7f41240f5830 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:46.751 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.748+0000 7f4128f57700 1 -- 172.21.15.184:0/3541580018 >> 172.21.15.184:0/3541580018 conn(0x7f41240f0b70 msgr2=0x7f41240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:46.751 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.748+0000 7f4128f57700 1 -- 172.21.15.184:0/3541580018 shutdown_connections 2024-02-19T22:15:46.751 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.748+0000 7f4128f57700 1 -- 172.21.15.184:0/3541580018 wait complete. 2024-02-19T22:15:46.752 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.748+0000 7f4128f57700 1 Processor -- start 2024-02-19T22:15:46.752 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.748+0000 7f4128f57700 1 -- start start 2024-02-19T22:15:46.752 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.748+0000 7f4128f57700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41240f5410 0x7f412410b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:46.752 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.748+0000 7f4128f57700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41241072f0 con 0x7f41240f5410 2024-02-19T22:15:46.752 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.749+0000 7f412259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41240f5410 0x7f412410b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:46.752 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.749+0000 7f412259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41240f5410 0x7f412410b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:40728/0 (socket says 172.21.15.184:40728) 2024-02-19T22:15:46.752 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.749+0000 7f412259c700 1 -- 172.21.15.184:0/673377024 learned_addr learned my addr 172.21.15.184:0/673377024 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:46.752 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.749+0000 7f412259c700 1 -- 172.21.15.184:0/673377024 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4114005040 con 0x7f41240f5410 2024-02-19T22:15:46.752 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.749+0000 7f412259c700 1 --2- 172.21.15.184:0/673377024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41240f5410 0x7f412410b900 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f4114005010 tx=0x7f411400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:46.752 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.749+0000 7f4112ffd700 1 -- 172.21.15.184:0/673377024 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4114009320 con 0x7f41240f5410 2024-02-19T22:15:46.752 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.749+0000 7f4128f57700 1 -- 172.21.15.184:0/673377024 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4124060220 con 0x7f41240f5410 2024-02-19T22:15:46.752 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.749+0000 7f4112ffd700 1 -- 172.21.15.184:0/673377024 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4114013380 con 0x7f41240f5410 2024-02-19T22:15:46.752 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.749+0000 7f4112ffd700 1 -- 172.21.15.184:0/673377024 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4114014e50 con 0x7f41240f5410 2024-02-19T22:15:46.752 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.749+0000 7f4128f57700 1 -- 172.21.15.184:0/673377024 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f412410bdf0 con 0x7f41240f5410 2024-02-19T22:15:46.753 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.750+0000 7f4112ffd700 1 -- 172.21.15.184:0/673377024 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41140095f0 con 0x7f41240f5410 2024-02-19T22:15:46.753 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.750+0000 7f4128f57700 1 -- 172.21.15.184:0/673377024 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4124045ad0 con 0x7f41240f5410 2024-02-19T22:15:46.753 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.750+0000 7f4112ffd700 1 --2- 172.21.15.184:0/673377024 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f410c02ff10 0x7f410c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:46.754 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.750+0000 7f4112ffd700 1 -- 172.21.15.184:0/673377024 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4114040fa0 con 0x7f41240f5410 2024-02-19T22:15:46.754 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.750+0000 7f4121d9b700 1 --2- 172.21.15.184:0/673377024 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f410c02ff10 0x7f410c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:46.754 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.751+0000 7f4121d9b700 1 --2- 172.21.15.184:0/673377024 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f410c02ff10 0x7f410c0323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f41180096a0 tx=0x7f4118006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:46.757 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.755+0000 7f4112ffd700 1 -- 172.21.15.184:0/673377024 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4114014420 con 0x7f41240f5410 2024-02-19T22:15:46.962 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.960+0000 7f4128f57700 1 -- 172.21.15.184:0/673377024 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f412410c5b0 con 0x7f41240f5410 2024-02-19T22:15:46.962 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.960+0000 7f4112ffd700 1 -- 172.21.15.184:0/673377024 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4114010230 con 0x7f41240f5410 2024-02-19T22:15:46.963 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:46.963 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:46.964 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.962+0000 7f4128f57700 1 -- 172.21.15.184:0/673377024 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f410c02ff10 msgr2=0x7f410c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:46.964 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.962+0000 7f4128f57700 1 --2- 172.21.15.184:0/673377024 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f410c02ff10 0x7f410c0323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f41180096a0 tx=0x7f4118006b40).stop 2024-02-19T22:15:46.964 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.962+0000 7f4128f57700 1 -- 172.21.15.184:0/673377024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41240f5410 msgr2=0x7f412410b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:46.964 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.962+0000 7f4128f57700 1 --2- 172.21.15.184:0/673377024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41240f5410 0x7f412410b900 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f4114005010 tx=0x7f411400b040).stop 2024-02-19T22:15:46.964 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.962+0000 7f4128f57700 1 -- 172.21.15.184:0/673377024 shutdown_connections 2024-02-19T22:15:46.964 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.962+0000 7f4128f57700 1 --2- 172.21.15.184:0/673377024 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f410c02ff10 0x7f410c0323d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:46.964 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.962+0000 7f4128f57700 1 --2- 172.21.15.184:0/673377024 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f41240f5410 0x7f412410b900 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:46.964 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.962+0000 7f4128f57700 1 -- 172.21.15.184:0/673377024 >> 172.21.15.184:0/673377024 conn(0x7f41240f0b70 msgr2=0x7f41240f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:46.964 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.962+0000 7f4128f57700 1 -- 172.21.15.184:0/673377024 shutdown_connections 2024-02-19T22:15:46.965 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:46.962+0000 7f4128f57700 1 -- 172.21.15.184:0/673377024 wait complete. 2024-02-19T22:15:46.965 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:48.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:45.895969+0000 mgr.smithi176.faoogw (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:48.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:46.961911+0000 mon.smithi176 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.184:0/673377024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:48.730 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:48.731 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:49.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:48 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:47.896307+0000 mgr.smithi176.faoogw (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:50.681 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.679+0000 7f784ad55700 1 -- 172.21.15.184:0/1078307344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78440f6510 msgr2=0x7f78440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:50.681 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.679+0000 7f784ad55700 1 --2- 172.21.15.184:0/1078307344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78440f6510 0x7f78440f6930 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f7838009fb0 tx=0x7f7838009b30).stop 2024-02-19T22:15:50.681 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.679+0000 7f784ad55700 1 -- 172.21.15.184:0/1078307344 shutdown_connections 2024-02-19T22:15:50.682 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.679+0000 7f784ad55700 1 --2- 172.21.15.184:0/1078307344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78440f6510 0x7f78440f6930 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:50.682 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.679+0000 7f784ad55700 1 -- 172.21.15.184:0/1078307344 >> 172.21.15.184:0/1078307344 conn(0x7f78440f1c70 msgr2=0x7f78440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:50.682 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.679+0000 7f784ad55700 1 -- 172.21.15.184:0/1078307344 shutdown_connections 2024-02-19T22:15:50.682 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.679+0000 7f784ad55700 1 -- 172.21.15.184:0/1078307344 wait complete. 2024-02-19T22:15:50.682 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.680+0000 7f784ad55700 1 Processor -- start 2024-02-19T22:15:50.682 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.680+0000 7f784ad55700 1 -- start start 2024-02-19T22:15:50.683 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.680+0000 7f784ad55700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78440f6510 0x7f784406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:50.683 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.680+0000 7f784ad55700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7844069e20 con 0x7f78440f6510 2024-02-19T22:15:50.683 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.680+0000 7f7848af1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78440f6510 0x7f784406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:50.683 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.681+0000 7f7848af1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78440f6510 0x7f784406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:40760/0 (socket says 172.21.15.184:40760) 2024-02-19T22:15:50.683 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.681+0000 7f7848af1700 1 -- 172.21.15.184:0/3386968018 learned_addr learned my addr 172.21.15.184:0/3386968018 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:50.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.681+0000 7f7848af1700 1 -- 172.21.15.184:0/3386968018 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7838005040 con 0x7f78440f6510 2024-02-19T22:15:50.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.681+0000 7f7848af1700 1 --2- 172.21.15.184:0/3386968018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78440f6510 0x7f784406b9a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f7838005010 tx=0x7f783800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:50.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.681+0000 7f78417fa700 1 -- 172.21.15.184:0/3386968018 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7838009320 con 0x7f78440f6510 2024-02-19T22:15:50.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.681+0000 7f784ad55700 1 -- 172.21.15.184:0/3386968018 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7844069fc0 con 0x7f78440f6510 2024-02-19T22:15:50.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.681+0000 7f78417fa700 1 -- 172.21.15.184:0/3386968018 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7838013380 con 0x7f78440f6510 2024-02-19T22:15:50.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.681+0000 7f78417fa700 1 -- 172.21.15.184:0/3386968018 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7838014e50 con 0x7f78440f6510 2024-02-19T22:15:50.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.681+0000 7f784ad55700 1 -- 172.21.15.184:0/3386968018 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f784406a450 con 0x7f78440f6510 2024-02-19T22:15:50.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.682+0000 7f78417fa700 1 -- 172.21.15.184:0/3386968018 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78380095f0 con 0x7f78440f6510 2024-02-19T22:15:50.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.682+0000 7f784ad55700 1 -- 172.21.15.184:0/3386968018 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7844059100 con 0x7f78440f6510 2024-02-19T22:15:50.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.682+0000 7f78417fa700 1 --2- 172.21.15.184:0/3386968018 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f783402ff10 0x7f78340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:50.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.682+0000 7f78417fa700 1 -- 172.21.15.184:0/3386968018 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7838040e70 con 0x7f78440f6510 2024-02-19T22:15:50.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.682+0000 7f7843fff700 1 --2- 172.21.15.184:0/3386968018 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f783402ff10 0x7f78340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:50.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.683+0000 7f7843fff700 1 --2- 172.21.15.184:0/3386968018 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f783402ff10 0x7f78340323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f78300096a0 tx=0x7f7830006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:50.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.687+0000 7f78417fa700 1 -- 172.21.15.184:0/3386968018 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7838014420 con 0x7f78440f6510 2024-02-19T22:15:50.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.892+0000 7f784ad55700 1 -- 172.21.15.184:0/3386968018 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78440fc0a0 con 0x7f78440f6510 2024-02-19T22:15:50.894 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.893+0000 7f78417fa700 1 -- 172.21.15.184:0/3386968018 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7838010230 con 0x7f78440f6510 2024-02-19T22:15:50.895 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:50.895 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:50.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.894+0000 7f784ad55700 1 -- 172.21.15.184:0/3386968018 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f783402ff10 msgr2=0x7f78340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:50.896 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.894+0000 7f784ad55700 1 --2- 172.21.15.184:0/3386968018 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f783402ff10 0x7f78340323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f78300096a0 tx=0x7f7830006b40).stop 2024-02-19T22:15:50.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.894+0000 7f784ad55700 1 -- 172.21.15.184:0/3386968018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78440f6510 msgr2=0x7f784406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:50.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.894+0000 7f784ad55700 1 --2- 172.21.15.184:0/3386968018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78440f6510 0x7f784406b9a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f7838005010 tx=0x7f783800b040).stop 2024-02-19T22:15:50.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.894+0000 7f784ad55700 1 -- 172.21.15.184:0/3386968018 shutdown_connections 2024-02-19T22:15:50.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.894+0000 7f784ad55700 1 --2- 172.21.15.184:0/3386968018 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f783402ff10 0x7f78340323d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:50.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.894+0000 7f784ad55700 1 --2- 172.21.15.184:0/3386968018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78440f6510 0x7f784406b9a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:50.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.894+0000 7f784ad55700 1 -- 172.21.15.184:0/3386968018 >> 172.21.15.184:0/3386968018 conn(0x7f78440f1c70 msgr2=0x7f78440f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:50.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.894+0000 7f784ad55700 1 -- 172.21.15.184:0/3386968018 shutdown_connections 2024-02-19T22:15:50.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:50.894+0000 7f784ad55700 1 -- 172.21.15.184:0/3386968018 wait complete. 2024-02-19T22:15:50.898 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:52.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:51 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:49.896593+0000 mgr.smithi176.faoogw (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:52.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:51 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:50.893845+0000 mon.smithi176 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.184:0/3386968018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:52.656 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:52.657 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:54.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:51.896869+0000 mgr.smithi176.faoogw (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:54.474 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.471+0000 7ff03e2dd700 1 -- 172.21.15.184:0/3112974151 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0380f5410 msgr2=0x7ff0380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:54.474 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.471+0000 7ff03e2dd700 1 --2- 172.21.15.184:0/3112974151 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0380f5410 0x7ff0380f5830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7ff028009fb0 tx=0x7ff028009b30).stop 2024-02-19T22:15:54.474 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.471+0000 7ff03e2dd700 1 -- 172.21.15.184:0/3112974151 shutdown_connections 2024-02-19T22:15:54.474 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.471+0000 7ff03e2dd700 1 --2- 172.21.15.184:0/3112974151 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0380f5410 0x7ff0380f5830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:54.474 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.471+0000 7ff03e2dd700 1 -- 172.21.15.184:0/3112974151 >> 172.21.15.184:0/3112974151 conn(0x7ff0380f0b70 msgr2=0x7ff0380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:54.474 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.471+0000 7ff03e2dd700 1 -- 172.21.15.184:0/3112974151 shutdown_connections 2024-02-19T22:15:54.474 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.471+0000 7ff03e2dd700 1 -- 172.21.15.184:0/3112974151 wait complete. 2024-02-19T22:15:54.474 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.472+0000 7ff03e2dd700 1 Processor -- start 2024-02-19T22:15:54.474 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.472+0000 7ff03e2dd700 1 -- start start 2024-02-19T22:15:54.474 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.472+0000 7ff03e2dd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0380fd520 0x7ff0380fb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:54.474 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.472+0000 7ff03e2dd700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0380fd940 con 0x7ff0380fd520 2024-02-19T22:15:54.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.472+0000 7ff037fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0380fd520 0x7ff0380fb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:54.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.472+0000 7ff037fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0380fd520 0x7ff0380fb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:40776/0 (socket says 172.21.15.184:40776) 2024-02-19T22:15:54.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.472+0000 7ff037fff700 1 -- 172.21.15.184:0/1659340236 learned_addr learned my addr 172.21.15.184:0/1659340236 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:54.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.473+0000 7ff037fff700 1 -- 172.21.15.184:0/1659340236 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff028005040 con 0x7ff0380fd520 2024-02-19T22:15:54.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.473+0000 7ff037fff700 1 --2- 172.21.15.184:0/1659340236 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0380fd520 0x7ff0380fb9e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7ff0380f6510 tx=0x7ff028009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:54.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.473+0000 7ff034ff9700 1 -- 172.21.15.184:0/1659340236 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff028013380 con 0x7ff0380fd520 2024-02-19T22:15:54.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.473+0000 7ff034ff9700 1 -- 172.21.15.184:0/1659340236 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff028013790 con 0x7ff0380fd520 2024-02-19T22:15:54.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.473+0000 7ff03e2dd700 1 -- 172.21.15.184:0/1659340236 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0380fbed0 con 0x7ff0380fd520 2024-02-19T22:15:54.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.473+0000 7ff034ff9700 1 -- 172.21.15.184:0/1659340236 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff02801ee70 con 0x7ff0380fd520 2024-02-19T22:15:54.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.473+0000 7ff03e2dd700 1 -- 172.21.15.184:0/1659340236 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0380fc3c0 con 0x7ff0380fd520 2024-02-19T22:15:54.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.474+0000 7ff034ff9700 1 -- 172.21.15.184:0/1659340236 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff028025060 con 0x7ff0380fd520 2024-02-19T22:15:54.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.474+0000 7ff03e2dd700 1 -- 172.21.15.184:0/1659340236 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff038045ad0 con 0x7ff0380fd520 2024-02-19T22:15:54.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.474+0000 7ff034ff9700 1 --2- 172.21.15.184:0/1659340236 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff02402ff60 0x7ff024032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:54.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.474+0000 7ff034ff9700 1 -- 172.21.15.184:0/1659340236 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff028040e50 con 0x7ff0380fd520 2024-02-19T22:15:54.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.474+0000 7ff0377fe700 1 --2- 172.21.15.184:0/1659340236 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff02402ff60 0x7ff024032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:54.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.474+0000 7ff0377fe700 1 --2- 172.21.15.184:0/1659340236 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff02402ff60 0x7ff024032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7ff0200096a0 tx=0x7ff020006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:54.481 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.478+0000 7ff034ff9700 1 -- 172.21.15.184:0/1659340236 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff02800f280 con 0x7ff0380fd520 2024-02-19T22:15:54.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.685+0000 7ff03e2dd700 1 -- 172.21.15.184:0/1659340236 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff03810cb00 con 0x7ff0380fd520 2024-02-19T22:15:54.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.685+0000 7ff034ff9700 1 -- 172.21.15.184:0/1659340236 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff02802d2d0 con 0x7ff0380fd520 2024-02-19T22:15:54.687 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:54.687 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:54.689 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.687+0000 7ff03e2dd700 1 -- 172.21.15.184:0/1659340236 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff02402ff60 msgr2=0x7ff024032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:54.689 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.687+0000 7ff03e2dd700 1 --2- 172.21.15.184:0/1659340236 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff02402ff60 0x7ff024032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7ff0200096a0 tx=0x7ff020006b40).stop 2024-02-19T22:15:54.689 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.687+0000 7ff03e2dd700 1 -- 172.21.15.184:0/1659340236 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0380fd520 msgr2=0x7ff0380fb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:54.689 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.687+0000 7ff03e2dd700 1 --2- 172.21.15.184:0/1659340236 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0380fd520 0x7ff0380fb9e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7ff0380f6510 tx=0x7ff028009b30).stop 2024-02-19T22:15:54.689 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.687+0000 7ff03e2dd700 1 -- 172.21.15.184:0/1659340236 shutdown_connections 2024-02-19T22:15:54.689 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.687+0000 7ff03e2dd700 1 --2- 172.21.15.184:0/1659340236 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff02402ff60 0x7ff024032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:54.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.687+0000 7ff03e2dd700 1 --2- 172.21.15.184:0/1659340236 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0380fd520 0x7ff0380fb9e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:54.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.687+0000 7ff03e2dd700 1 -- 172.21.15.184:0/1659340236 >> 172.21.15.184:0/1659340236 conn(0x7ff0380f0b70 msgr2=0x7ff03805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:54.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.687+0000 7ff03e2dd700 1 -- 172.21.15.184:0/1659340236 shutdown_connections 2024-02-19T22:15:54.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:54.687+0000 7ff03e2dd700 1 -- 172.21.15.184:0/1659340236 wait complete. 2024-02-19T22:15:54.690 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:55.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:54 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:54.686813+0000 mon.smithi176 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.184:0/1659340236' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:56.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:55 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:53.897193+0000 mgr.smithi176.faoogw (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:56.512 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:56.513 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:55.897586+0000 mgr.smithi176.faoogw (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:58.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.450+0000 7f9fdf59e700 1 -- 172.21.15.184:0/3205871210 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9fe00f51a0 msgr2=0x7f9fe00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:58.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.451+0000 7f9fdf59e700 1 --2- 172.21.15.184:0/3205871210 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9fe00f51a0 0x7f9fe00f55c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f9fd0009960 tx=0x7f9fd0004d10).stop 2024-02-19T22:15:58.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.451+0000 7f9fdf59e700 1 -- 172.21.15.184:0/3205871210 shutdown_connections 2024-02-19T22:15:58.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.451+0000 7f9fdf59e700 1 --2- 172.21.15.184:0/3205871210 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9fe00f51a0 0x7f9fe00f55c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:58.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.451+0000 7f9fdf59e700 1 -- 172.21.15.184:0/3205871210 >> 172.21.15.184:0/3205871210 conn(0x7f9fe00f09a0 msgr2=0x7f9fe00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:58.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.451+0000 7f9fdf59e700 1 -- 172.21.15.184:0/3205871210 shutdown_connections 2024-02-19T22:15:58.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.451+0000 7f9fdf59e700 1 -- 172.21.15.184:0/3205871210 wait complete. 2024-02-19T22:15:58.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.451+0000 7f9fdf59e700 1 Processor -- start 2024-02-19T22:15:58.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.451+0000 7f9fdf59e700 1 -- start start 2024-02-19T22:15:58.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.451+0000 7f9fdf59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9fe00f51a0 0x7f9fe01065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:58.454 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.451+0000 7f9fdf59e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe0106ac0 con 0x7f9fe00f51a0 2024-02-19T22:15:58.455 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.451+0000 7f9fde59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9fe00f51a0 0x7f9fe01065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:58.455 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.451+0000 7f9fde59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9fe00f51a0 0x7f9fe01065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:52454/0 (socket says 172.21.15.184:52454) 2024-02-19T22:15:58.455 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.451+0000 7f9fde59c700 1 -- 172.21.15.184:0/555325900 learned_addr learned my addr 172.21.15.184:0/555325900 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:15:58.455 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.452+0000 7f9fde59c700 1 -- 172.21.15.184:0/555325900 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd0009640 con 0x7f9fe00f51a0 2024-02-19T22:15:58.455 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.452+0000 7f9fde59c700 1 --2- 172.21.15.184:0/555325900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9fe00f51a0 0x7f9fe01065d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f9fd0007bd0 tx=0x7f9fd0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:58.455 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.452+0000 7f9fd6ffd700 1 -- 172.21.15.184:0/555325900 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fd0018070 con 0x7f9fe00f51a0 2024-02-19T22:15:58.455 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.452+0000 7f9fdf59e700 1 -- 172.21.15.184:0/555325900 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe0106cc0 con 0x7f9fe00f51a0 2024-02-19T22:15:58.455 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.452+0000 7f9fd6ffd700 1 -- 172.21.15.184:0/555325900 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9fd000b970 con 0x7f9fe00f51a0 2024-02-19T22:15:58.455 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.452+0000 7f9fd6ffd700 1 -- 172.21.15.184:0/555325900 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fd001d600 con 0x7f9fe00f51a0 2024-02-19T22:15:58.455 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.452+0000 7f9fdf59e700 1 -- 172.21.15.184:0/555325900 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe0107150 con 0x7f9fe00f51a0 2024-02-19T22:15:58.456 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.453+0000 7f9fd6ffd700 1 -- 172.21.15.184:0/555325900 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9fd0024060 con 0x7f9fe00f51a0 2024-02-19T22:15:58.457 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.453+0000 7f9fdf59e700 1 -- 172.21.15.184:0/555325900 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe0045ad0 con 0x7f9fe00f51a0 2024-02-19T22:15:58.457 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.453+0000 7f9fd6ffd700 1 --2- 172.21.15.184:0/555325900 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9fcc02ff60 0x7f9fcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:58.457 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.453+0000 7f9fd6ffd700 1 -- 172.21.15.184:0/555325900 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9fd00414e0 con 0x7f9fe00f51a0 2024-02-19T22:15:58.457 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.453+0000 7f9fddd9b700 1 --2- 172.21.15.184:0/555325900 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9fcc02ff60 0x7f9fcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:58.457 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.453+0000 7f9fddd9b700 1 --2- 172.21.15.184:0/555325900 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9fcc02ff60 0x7f9fcc032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9fc80096a0 tx=0x7f9fc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:58.459 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.458+0000 7f9fd6ffd700 1 -- 172.21.15.184:0/555325900 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9fd0013810 con 0x7f9fe00f51a0 2024-02-19T22:15:58.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.660+0000 7f9fdf59e700 1 -- 172.21.15.184:0/555325900 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9fe0108000 con 0x7f9fe00f51a0 2024-02-19T22:15:58.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.661+0000 7f9fd6ffd700 1 -- 172.21.15.184:0/555325900 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9fd0013630 con 0x7f9fe00f51a0 2024-02-19T22:15:58.664 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:15:58.664 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:58.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.663+0000 7f9fdf59e700 1 -- 172.21.15.184:0/555325900 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9fcc02ff60 msgr2=0x7f9fcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:58.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.663+0000 7f9fdf59e700 1 --2- 172.21.15.184:0/555325900 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9fcc02ff60 0x7f9fcc032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9fc80096a0 tx=0x7f9fc8006b40).stop 2024-02-19T22:15:58.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.663+0000 7f9fdf59e700 1 -- 172.21.15.184:0/555325900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9fe00f51a0 msgr2=0x7f9fe01065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:58.665 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.663+0000 7f9fdf59e700 1 --2- 172.21.15.184:0/555325900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9fe00f51a0 0x7f9fe01065d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f9fd0007bd0 tx=0x7f9fd0004d10).stop 2024-02-19T22:15:58.666 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.663+0000 7f9fdf59e700 1 -- 172.21.15.184:0/555325900 shutdown_connections 2024-02-19T22:15:58.666 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.663+0000 7f9fdf59e700 1 --2- 172.21.15.184:0/555325900 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9fcc02ff60 0x7f9fcc032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:58.666 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.663+0000 7f9fdf59e700 1 --2- 172.21.15.184:0/555325900 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9fe00f51a0 0x7f9fe01065d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:58.666 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.663+0000 7f9fdf59e700 1 -- 172.21.15.184:0/555325900 >> 172.21.15.184:0/555325900 conn(0x7f9fe00f09a0 msgr2=0x7f9fe005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:58.666 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.663+0000 7f9fdf59e700 1 -- 172.21.15.184:0/555325900 shutdown_connections 2024-02-19T22:15:58.666 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:15:58.663+0000 7f9fdf59e700 1 -- 172.21.15.184:0/555325900 wait complete. 2024-02-19T22:15:58.666 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:15:59.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:58 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:15:58.662599+0000 mon.smithi176 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.184:0/555325900' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:00.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:15:59 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:57.897940+0000 mgr.smithi176.faoogw (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:00.327 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:00.327 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:02.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:15:59.898293+0000 mgr.smithi176.faoogw (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:02.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.467+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1428287138 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfa0060280 msgr2=0x7fbfa00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:02.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.467+0000 7fbfa7d46700 1 --2- 172.21.15.184:0/1428287138 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfa0060280 0x7fbfa00f90f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fbf90009fb0 tx=0x7fbf90009b30).stop 2024-02-19T22:16:02.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.468+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1428287138 shutdown_connections 2024-02-19T22:16:02.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.468+0000 7fbfa7d46700 1 --2- 172.21.15.184:0/1428287138 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfa0060280 0x7fbfa00f90f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:02.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.468+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1428287138 >> 172.21.15.184:0/1428287138 conn(0x7fbfa00f0b70 msgr2=0x7fbfa00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:02.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.468+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1428287138 shutdown_connections 2024-02-19T22:16:02.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.468+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1428287138 wait complete. 2024-02-19T22:16:02.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.468+0000 7fbfa7d46700 1 Processor -- start 2024-02-19T22:16:02.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.468+0000 7fbfa7d46700 1 -- start start 2024-02-19T22:16:02.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.468+0000 7fbfa7d46700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfa0060280 0x7fbfa00f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:02.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.468+0000 7fbfa7d46700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa00f4ff0 con 0x7fbfa0060280 2024-02-19T22:16:02.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.469+0000 7fbfa5ae2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfa0060280 0x7fbfa00f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:02.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.469+0000 7fbfa5ae2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfa0060280 0x7fbfa00f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:52476/0 (socket says 172.21.15.184:52476) 2024-02-19T22:16:02.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.469+0000 7fbfa5ae2700 1 -- 172.21.15.184:0/1110329322 learned_addr learned my addr 172.21.15.184:0/1110329322 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:02.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.469+0000 7fbfa5ae2700 1 -- 172.21.15.184:0/1110329322 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf90005040 con 0x7fbfa0060280 2024-02-19T22:16:02.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.469+0000 7fbfa5ae2700 1 --2- 172.21.15.184:0/1110329322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfa0060280 0x7fbfa00f6b70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fbf90009830 tx=0x7fbf9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:02.472 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.469+0000 7fbf967fc700 1 -- 172.21.15.184:0/1110329322 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf90009320 con 0x7fbfa0060280 2024-02-19T22:16:02.472 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.469+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1110329322 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa00f5190 con 0x7fbfa0060280 2024-02-19T22:16:02.472 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.469+0000 7fbf967fc700 1 -- 172.21.15.184:0/1110329322 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf90013380 con 0x7fbfa0060280 2024-02-19T22:16:02.472 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.469+0000 7fbf967fc700 1 -- 172.21.15.184:0/1110329322 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf90014420 con 0x7fbfa0060280 2024-02-19T22:16:02.472 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.469+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1110329322 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa00f5620 con 0x7fbfa0060280 2024-02-19T22:16:02.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.470+0000 7fbf967fc700 1 -- 172.21.15.184:0/1110329322 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbf9001e420 con 0x7fbfa0060280 2024-02-19T22:16:02.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.470+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1110329322 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa0059100 con 0x7fbfa0060280 2024-02-19T22:16:02.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.470+0000 7fbf967fc700 1 --2- 172.21.15.184:0/1110329322 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbf8c02ff60 0x7fbf8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:02.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.470+0000 7fbf967fc700 1 -- 172.21.15.184:0/1110329322 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbf90040ee0 con 0x7fbfa0060280 2024-02-19T22:16:02.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.471+0000 7fbfa52e1700 1 --2- 172.21.15.184:0/1110329322 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbf8c02ff60 0x7fbf8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:02.473 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.471+0000 7fbfa52e1700 1 --2- 172.21.15.184:0/1110329322 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbf8c02ff60 0x7fbf8c032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fbf9c0096a0 tx=0x7fbf9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:02.475 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.475+0000 7fbf967fc700 1 -- 172.21.15.184:0/1110329322 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf9001c930 con 0x7fbfa0060280 2024-02-19T22:16:02.683 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.683+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1110329322 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbfa00fc1e0 con 0x7fbfa0060280 2024-02-19T22:16:02.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.683+0000 7fbf967fc700 1 -- 172.21.15.184:0/1110329322 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbf9002c2d0 con 0x7fbfa0060280 2024-02-19T22:16:02.684 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:02.684 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:02.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.685+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1110329322 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbf8c02ff60 msgr2=0x7fbf8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:02.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.685+0000 7fbfa7d46700 1 --2- 172.21.15.184:0/1110329322 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbf8c02ff60 0x7fbf8c032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fbf9c0096a0 tx=0x7fbf9c006b40).stop 2024-02-19T22:16:02.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.685+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1110329322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfa0060280 msgr2=0x7fbfa00f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:02.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.685+0000 7fbfa7d46700 1 --2- 172.21.15.184:0/1110329322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfa0060280 0x7fbfa00f6b70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fbf90009830 tx=0x7fbf9000b040).stop 2024-02-19T22:16:02.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.685+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1110329322 shutdown_connections 2024-02-19T22:16:02.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.685+0000 7fbfa7d46700 1 --2- 172.21.15.184:0/1110329322 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbf8c02ff60 0x7fbf8c032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:02.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.685+0000 7fbfa7d46700 1 --2- 172.21.15.184:0/1110329322 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbfa0060280 0x7fbfa00f6b70 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:02.686 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.685+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1110329322 >> 172.21.15.184:0/1110329322 conn(0x7fbfa00f0b70 msgr2=0x7fbfa00f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:02.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.685+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1110329322 shutdown_connections 2024-02-19T22:16:02.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:02.685+0000 7fbfa7d46700 1 -- 172.21.15.184:0/1110329322 wait complete. 2024-02-19T22:16:02.687 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:03.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:02 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:02.683722+0000 mon.smithi176 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.184:0/1110329322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:04.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:01.898654+0000 mgr.smithi176.faoogw (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:04.354 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:04.354 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:06.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:03.898918+0000 mgr.smithi176.faoogw (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:06.209 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.207+0000 7fa23e6b9700 1 -- 172.21.15.184:0/3094368357 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2380f51a0 msgr2=0x7fa2380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:06.209 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.207+0000 7fa23e6b9700 1 --2- 172.21.15.184:0/3094368357 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2380f51a0 0x7fa2380f55c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fa234009fb0 tx=0x7fa234009b30).stop 2024-02-19T22:16:06.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.207+0000 7fa23e6b9700 1 -- 172.21.15.184:0/3094368357 shutdown_connections 2024-02-19T22:16:06.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.207+0000 7fa23e6b9700 1 --2- 172.21.15.184:0/3094368357 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2380f51a0 0x7fa2380f55c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:06.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.207+0000 7fa23e6b9700 1 -- 172.21.15.184:0/3094368357 >> 172.21.15.184:0/3094368357 conn(0x7fa2380f09a0 msgr2=0x7fa2380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:06.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.207+0000 7fa23e6b9700 1 -- 172.21.15.184:0/3094368357 shutdown_connections 2024-02-19T22:16:06.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.207+0000 7fa23e6b9700 1 -- 172.21.15.184:0/3094368357 wait complete. 2024-02-19T22:16:06.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.207+0000 7fa23e6b9700 1 Processor -- start 2024-02-19T22:16:06.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.207+0000 7fa23e6b9700 1 -- start start 2024-02-19T22:16:06.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.208+0000 7fa23e6b9700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2380f51a0 0x7fa2381092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:06.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.208+0000 7fa23e6b9700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa238060220 con 0x7fa2380f51a0 2024-02-19T22:16:06.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.208+0000 7fa23d6b7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2380f51a0 0x7fa2381092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:06.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.208+0000 7fa23d6b7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2380f51a0 0x7fa2381092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:52500/0 (socket says 172.21.15.184:52500) 2024-02-19T22:16:06.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.208+0000 7fa23d6b7700 1 -- 172.21.15.184:0/197275034 learned_addr learned my addr 172.21.15.184:0/197275034 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:06.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.208+0000 7fa23d6b7700 1 -- 172.21.15.184:0/197275034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa234005040 con 0x7fa2380f51a0 2024-02-19T22:16:06.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.208+0000 7fa23d6b7700 1 --2- 172.21.15.184:0/197275034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2380f51a0 0x7fa2381092f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fa2340089f0 tx=0x7fa23400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:06.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.208+0000 7fa22dffb700 1 -- 172.21.15.184:0/197275034 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa234009320 con 0x7fa2380f51a0 2024-02-19T22:16:06.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.208+0000 7fa23e6b9700 1 -- 172.21.15.184:0/197275034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa238060420 con 0x7fa2380f51a0 2024-02-19T22:16:06.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.208+0000 7fa22dffb700 1 -- 172.21.15.184:0/197275034 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa234013380 con 0x7fa2380f51a0 2024-02-19T22:16:06.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.208+0000 7fa22dffb700 1 -- 172.21.15.184:0/197275034 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa234014e20 con 0x7fa2380f51a0 2024-02-19T22:16:06.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.209+0000 7fa23e6b9700 1 -- 172.21.15.184:0/197275034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2381099a0 con 0x7fa2380f51a0 2024-02-19T22:16:06.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.209+0000 7fa22dffb700 1 -- 172.21.15.184:0/197275034 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2340095f0 con 0x7fa2380f51a0 2024-02-19T22:16:06.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.209+0000 7fa23e6b9700 1 -- 172.21.15.184:0/197275034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa238059100 con 0x7fa2380f51a0 2024-02-19T22:16:06.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.210+0000 7fa22dffb700 1 --2- 172.21.15.184:0/197275034 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa22402ff10 0x7fa2240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:06.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.210+0000 7fa22dffb700 1 -- 172.21.15.184:0/197275034 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa234040f70 con 0x7fa2380f51a0 2024-02-19T22:16:06.213 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.210+0000 7fa23ceb6700 1 --2- 172.21.15.184:0/197275034 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa22402ff10 0x7fa2240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:06.213 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.210+0000 7fa23ceb6700 1 --2- 172.21.15.184:0/197275034 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa22402ff10 0x7fa2240323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fa2280096a0 tx=0x7fa228006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:06.216 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.214+0000 7fa22dffb700 1 -- 172.21.15.184:0/197275034 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa23406d2b0 con 0x7fa2380f51a0 2024-02-19T22:16:06.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.419+0000 7fa23e6b9700 1 -- 172.21.15.184:0/197275034 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2380fad30 con 0x7fa2380f51a0 2024-02-19T22:16:06.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.419+0000 7fa22dffb700 1 -- 172.21.15.184:0/197275034 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa234014420 con 0x7fa2380f51a0 2024-02-19T22:16:06.422 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:06.422 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:06.423 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.421+0000 7fa23e6b9700 1 -- 172.21.15.184:0/197275034 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa22402ff10 msgr2=0x7fa2240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:06.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.421+0000 7fa23e6b9700 1 --2- 172.21.15.184:0/197275034 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa22402ff10 0x7fa2240323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fa2280096a0 tx=0x7fa228006b40).stop 2024-02-19T22:16:06.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.421+0000 7fa23e6b9700 1 -- 172.21.15.184:0/197275034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2380f51a0 msgr2=0x7fa2381092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:06.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.421+0000 7fa23e6b9700 1 --2- 172.21.15.184:0/197275034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2380f51a0 0x7fa2381092f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fa2340089f0 tx=0x7fa23400b040).stop 2024-02-19T22:16:06.424 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.421+0000 7fa23e6b9700 1 -- 172.21.15.184:0/197275034 shutdown_connections 2024-02-19T22:16:06.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.421+0000 7fa23e6b9700 1 --2- 172.21.15.184:0/197275034 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa22402ff10 0x7fa2240323d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:06.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.421+0000 7fa23e6b9700 1 --2- 172.21.15.184:0/197275034 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2380f51a0 0x7fa2381092f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:06.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.421+0000 7fa23e6b9700 1 -- 172.21.15.184:0/197275034 >> 172.21.15.184:0/197275034 conn(0x7fa2380f09a0 msgr2=0x7fa2380f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:06.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.421+0000 7fa23e6b9700 1 -- 172.21.15.184:0/197275034 shutdown_connections 2024-02-19T22:16:06.425 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:06.421+0000 7fa23e6b9700 1 -- 172.21.15.184:0/197275034 wait complete. 2024-02-19T22:16:06.426 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:07.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:16:04.925769+0000 mgr.smithi176.faoogw (mgr.14162) 231 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi176 ... 2024-02-19T22:16:07.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Verifying port 9100 ... 2024-02-19T22:16:07.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Non-zero exit code 1 from systemctl start ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi176 2024-02-19T22:16:07.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: systemctl: stderr Job for ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi176.service failed because a timeout was exceeded. 2024-02-19T22:16:07.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: systemctl: stderr See "systemctl status ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi176.service" and "journalctl -xe" for details. 2024-02-19T22:16:07.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Traceback (most recent call last): 2024-02-19T22:16:07.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:16:07.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: main() 2024-02-19T22:16:07.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:16:07.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: r = ctx.func(ctx) 2024-02-19T22:16:07.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:16:07.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: return func(ctx) 2024-02-19T22:16:07.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-19T22:16:07.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: ports=daemon_ports) 2024-02-19T22:16:07.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-19T22:16:07.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-19T22:16:07.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-19T22:16:07.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-19T22:16:07.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-19T22:16:07.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-19T22:16:07.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: RuntimeError: Failed command: systemctl start ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi176 2024-02-19T22:16:07.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Traceback (most recent call last): 2024-02-19T22:16:07.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:16:07.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: yield (conn, connr) 2024-02-19T22:16:07.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:16:07.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: code, '\n'.join(err))) 2024-02-19T22:16:07.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi176 ... 2024-02-19T22:16:07.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Verifying port 9100 ... 2024-02-19T22:16:07.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Non-zero exit code 1 from systemctl start ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi176 2024-02-19T22:16:07.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: systemctl: stderr Job for ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi176.service failed because a timeout was exceeded. 2024-02-19T22:16:07.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: systemctl: stderr See "systemctl status ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi176.service" and "journalctl -xe" for details. 2024-02-19T22:16:07.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Traceback (most recent call last): 2024-02-19T22:16:07.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:16:07.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: main() 2024-02-19T22:16:07.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:16:07.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: r = ctx.func(ctx) 2024-02-19T22:16:07.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:16:07.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: return func(ctx) 2024-02-19T22:16:07.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-19T22:16:07.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: ports=daemon_ports) 2024-02-19T22:16:07.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-19T22:16:07.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-19T22:16:07.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-19T22:16:07.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-19T22:16:07.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-19T22:16:07.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-19T22:16:07.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: RuntimeError: Failed command: systemctl start ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi176 2024-02-19T22:16:07.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:16:04.932625+0000 mgr.smithi176.faoogw (mgr.14162) 232 : cephadm [INF] Deploying daemon prometheus.smithi176 on smithi176 2024-02-19T22:16:07.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:06 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:06.420790+0000 mon.smithi176 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.184:0/197275034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:08.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:05.899242+0000 mgr.smithi176.faoogw (mgr.14162) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:08.147 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:08.148 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:10.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.068+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/2291223424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c880601e0 msgr2=0x7f0c880f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:10.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.068+0000 7f0c8cfa4700 1 --2- 172.21.15.184:0/2291223424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c880601e0 0x7f0c880f9190 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f0c70009fb0 tx=0x7f0c70009b30).stop 2024-02-19T22:16:10.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.068+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/2291223424 shutdown_connections 2024-02-19T22:16:10.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.068+0000 7f0c8cfa4700 1 --2- 172.21.15.184:0/2291223424 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c880601e0 0x7f0c880f9190 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:10.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.068+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/2291223424 >> 172.21.15.184:0/2291223424 conn(0x7f0c880f0b70 msgr2=0x7f0c880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:10.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.068+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/2291223424 shutdown_connections 2024-02-19T22:16:10.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.068+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/2291223424 wait complete. 2024-02-19T22:16:10.071 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.068+0000 7f0c8cfa4700 1 Processor -- start 2024-02-19T22:16:10.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.068+0000 7f0c8cfa4700 1 -- start start 2024-02-19T22:16:10.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.068+0000 7f0c8cfa4700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c880601e0 0x7f0c880f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:10.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.068+0000 7f0c8cfa4700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c880f5090 con 0x7f0c880601e0 2024-02-19T22:16:10.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.069+0000 7f0c8659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c880601e0 0x7f0c880f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:10.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.069+0000 7f0c8659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c880601e0 0x7f0c880f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:45368/0 (socket says 172.21.15.184:45368) 2024-02-19T22:16:10.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.069+0000 7f0c8659c700 1 -- 172.21.15.184:0/430932728 learned_addr learned my addr 172.21.15.184:0/430932728 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:10.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.069+0000 7f0c8659c700 1 -- 172.21.15.184:0/430932728 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c70005040 con 0x7f0c880601e0 2024-02-19T22:16:10.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.069+0000 7f0c8659c700 1 --2- 172.21.15.184:0/430932728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c880601e0 0x7f0c880f6c10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f0c70007a10 tx=0x7f0c70009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:10.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.069+0000 7f0c7effd700 1 -- 172.21.15.184:0/430932728 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c70018070 con 0x7f0c880601e0 2024-02-19T22:16:10.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.069+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/430932728 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c880f5230 con 0x7f0c880601e0 2024-02-19T22:16:10.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.069+0000 7f0c7effd700 1 -- 172.21.15.184:0/430932728 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c70013410 con 0x7f0c880601e0 2024-02-19T22:16:10.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.069+0000 7f0c7effd700 1 -- 172.21.15.184:0/430932728 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c7001c3c0 con 0x7f0c880601e0 2024-02-19T22:16:10.072 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.069+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/430932728 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c880f56c0 con 0x7f0c880601e0 2024-02-19T22:16:10.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.070+0000 7f0c7effd700 1 -- 172.21.15.184:0/430932728 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c7001d420 con 0x7f0c880601e0 2024-02-19T22:16:10.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.070+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/430932728 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c88059100 con 0x7f0c880601e0 2024-02-19T22:16:10.073 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.070+0000 7f0c7effd700 1 --2- 172.21.15.184:0/430932728 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f0c7402ff60 0x7f0c74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:10.074 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.070+0000 7f0c7effd700 1 -- 172.21.15.184:0/430932728 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c700414b0 con 0x7f0c880601e0 2024-02-19T22:16:10.074 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.070+0000 7f0c85d9b700 1 --2- 172.21.15.184:0/430932728 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f0c7402ff60 0x7f0c74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:10.074 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.071+0000 7f0c85d9b700 1 --2- 172.21.15.184:0/430932728 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f0c7402ff60 0x7f0c74032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f0c780096a0 tx=0x7f0c78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:10.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.075+0000 7f0c7effd700 1 -- 172.21.15.184:0/430932728 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c7001d820 con 0x7f0c880601e0 2024-02-19T22:16:10.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:07.899605+0000 mgr.smithi176.faoogw (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:10.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.280+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/430932728 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c88045ad0 con 0x7f0c880601e0 2024-02-19T22:16:10.282 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.280+0000 7f0c7effd700 1 -- 172.21.15.184:0/430932728 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0c7001b370 con 0x7f0c880601e0 2024-02-19T22:16:10.282 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:10.282 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:10.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.282+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/430932728 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f0c7402ff60 msgr2=0x7f0c74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:10.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.282+0000 7f0c8cfa4700 1 --2- 172.21.15.184:0/430932728 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f0c7402ff60 0x7f0c74032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f0c780096a0 tx=0x7f0c78006b40).stop 2024-02-19T22:16:10.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.282+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/430932728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c880601e0 msgr2=0x7f0c880f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:10.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.282+0000 7f0c8cfa4700 1 --2- 172.21.15.184:0/430932728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c880601e0 0x7f0c880f6c10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f0c70007a10 tx=0x7f0c70009b30).stop 2024-02-19T22:16:10.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.282+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/430932728 shutdown_connections 2024-02-19T22:16:10.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.282+0000 7f0c8cfa4700 1 --2- 172.21.15.184:0/430932728 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f0c7402ff60 0x7f0c74032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:10.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.282+0000 7f0c8cfa4700 1 --2- 172.21.15.184:0/430932728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c880601e0 0x7f0c880f6c10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:10.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.282+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/430932728 >> 172.21.15.184:0/430932728 conn(0x7f0c880f0b70 msgr2=0x7f0c880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:10.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.282+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/430932728 shutdown_connections 2024-02-19T22:16:10.284 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:10.282+0000 7f0c8cfa4700 1 -- 172.21.15.184:0/430932728 wait complete. 2024-02-19T22:16:10.285 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:11.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:10 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:10.281681+0000 mon.smithi176 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.184:0/430932728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:12.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:12.043 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:12.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:09.899977+0000 mgr.smithi176.faoogw (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:13.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.939+0000 7fb7a97f3700 1 -- 172.21.15.184:0/722374764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a40f6510 msgr2=0x7fb7a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:13.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.939+0000 7fb7a97f3700 1 --2- 172.21.15.184:0/722374764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a40f6510 0x7fb7a40f6930 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fb78c009fb0 tx=0x7fb78c009b30).stop 2024-02-19T22:16:13.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.939+0000 7fb7a97f3700 1 -- 172.21.15.184:0/722374764 shutdown_connections 2024-02-19T22:16:13.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.939+0000 7fb7a97f3700 1 --2- 172.21.15.184:0/722374764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a40f6510 0x7fb7a40f6930 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:13.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.939+0000 7fb7a97f3700 1 -- 172.21.15.184:0/722374764 >> 172.21.15.184:0/722374764 conn(0x7fb7a40f1c70 msgr2=0x7fb7a40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:13.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.939+0000 7fb7a97f3700 1 -- 172.21.15.184:0/722374764 shutdown_connections 2024-02-19T22:16:13.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.939+0000 7fb7a97f3700 1 -- 172.21.15.184:0/722374764 wait complete. 2024-02-19T22:16:13.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.940+0000 7fb7a97f3700 1 Processor -- start 2024-02-19T22:16:13.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.940+0000 7fb7a97f3700 1 -- start start 2024-02-19T22:16:13.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.940+0000 7fb7a97f3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a40f6510 0x7fb7a406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:13.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.940+0000 7fb7a97f3700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4069e20 con 0x7fb7a40f6510 2024-02-19T22:16:13.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.940+0000 7fb7a2ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a40f6510 0x7fb7a406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:13.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.940+0000 7fb7a2ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a40f6510 0x7fb7a406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:45380/0 (socket says 172.21.15.184:45380) 2024-02-19T22:16:13.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.941+0000 7fb7a2ffd700 1 -- 172.21.15.184:0/1068014894 learned_addr learned my addr 172.21.15.184:0/1068014894 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:13.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.941+0000 7fb7a2ffd700 1 -- 172.21.15.184:0/1068014894 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c005040 con 0x7fb7a40f6510 2024-02-19T22:16:13.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.941+0000 7fb7a2ffd700 1 --2- 172.21.15.184:0/1068014894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a40f6510 0x7fb7a406b9a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fb78c005010 tx=0x7fb78c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:13.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.941+0000 7fb79b7fe700 1 -- 172.21.15.184:0/1068014894 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb78c009320 con 0x7fb7a40f6510 2024-02-19T22:16:13.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.941+0000 7fb7a97f3700 1 -- 172.21.15.184:0/1068014894 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a4069fc0 con 0x7fb7a40f6510 2024-02-19T22:16:13.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.941+0000 7fb79b7fe700 1 -- 172.21.15.184:0/1068014894 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb78c013380 con 0x7fb7a40f6510 2024-02-19T22:16:13.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.941+0000 7fb79b7fe700 1 -- 172.21.15.184:0/1068014894 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb78c014e50 con 0x7fb7a40f6510 2024-02-19T22:16:13.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.941+0000 7fb7a97f3700 1 -- 172.21.15.184:0/1068014894 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a406a450 con 0x7fb7a40f6510 2024-02-19T22:16:13.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.942+0000 7fb79b7fe700 1 -- 172.21.15.184:0/1068014894 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb78c014420 con 0x7fb7a40f6510 2024-02-19T22:16:13.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.942+0000 7fb7a97f3700 1 -- 172.21.15.184:0/1068014894 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a4059100 con 0x7fb7a40f6510 2024-02-19T22:16:13.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.942+0000 7fb79b7fe700 1 --2- 172.21.15.184:0/1068014894 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb79002ff10 0x7fb7900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:13.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.942+0000 7fb79b7fe700 1 -- 172.21.15.184:0/1068014894 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb78c040f70 con 0x7fb7a40f6510 2024-02-19T22:16:13.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.942+0000 7fb7a27fc700 1 --2- 172.21.15.184:0/1068014894 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb79002ff10 0x7fb7900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:13.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.943+0000 7fb7a27fc700 1 --2- 172.21.15.184:0/1068014894 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb79002ff10 0x7fb7900323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fb7940096a0 tx=0x7fb794006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:13.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:13.947+0000 7fb79b7fe700 1 -- 172.21.15.184:0/1068014894 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb78c0095f0 con 0x7fb7a40f6510 2024-02-19T22:16:14.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:13 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:11.900222+0000 mgr.smithi176.faoogw (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:14.152 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:14.150+0000 7fb7a97f3700 1 -- 172.21.15.184:0/1068014894 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb7a4045ad0 con 0x7fb7a40f6510 2024-02-19T22:16:14.153 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:14.151+0000 7fb79b7fe700 1 -- 172.21.15.184:0/1068014894 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb78c01db70 con 0x7fb7a40f6510 2024-02-19T22:16:14.153 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:14.153 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:14.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:14.153+0000 7fb7a97f3700 1 -- 172.21.15.184:0/1068014894 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb79002ff10 msgr2=0x7fb7900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:14.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:14.153+0000 7fb7a97f3700 1 --2- 172.21.15.184:0/1068014894 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb79002ff10 0x7fb7900323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fb7940096a0 tx=0x7fb794006b40).stop 2024-02-19T22:16:14.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:14.153+0000 7fb7a97f3700 1 -- 172.21.15.184:0/1068014894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a40f6510 msgr2=0x7fb7a406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:14.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:14.153+0000 7fb7a97f3700 1 --2- 172.21.15.184:0/1068014894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a40f6510 0x7fb7a406b9a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fb78c005010 tx=0x7fb78c00b040).stop 2024-02-19T22:16:14.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:14.153+0000 7fb7a97f3700 1 -- 172.21.15.184:0/1068014894 shutdown_connections 2024-02-19T22:16:14.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:14.153+0000 7fb7a97f3700 1 --2- 172.21.15.184:0/1068014894 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb79002ff10 0x7fb7900323d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:14.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:14.153+0000 7fb7a97f3700 1 --2- 172.21.15.184:0/1068014894 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb7a40f6510 0x7fb7a406b9a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:14.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:14.153+0000 7fb7a97f3700 1 -- 172.21.15.184:0/1068014894 >> 172.21.15.184:0/1068014894 conn(0x7fb7a40f1c70 msgr2=0x7fb7a40f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:14.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:14.153+0000 7fb7a97f3700 1 -- 172.21.15.184:0/1068014894 shutdown_connections 2024-02-19T22:16:14.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:14.153+0000 7fb7a97f3700 1 -- 172.21.15.184:0/1068014894 wait complete. 2024-02-19T22:16:14.156 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:15.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:14 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:14.152498+0000 mon.smithi176 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.184:0/1068014894' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:15.825 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:15.825 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:16.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:13.900583+0000 mgr.smithi176.faoogw (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:17.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.722+0000 7fd97a1fa700 1 -- 172.21.15.184:0/605397361 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9740f4740 msgr2=0x7fd9740f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:17.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.722+0000 7fd97a1fa700 1 --2- 172.21.15.184:0/605397361 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9740f4740 0x7fd9740f4b60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fd964009fb0 tx=0x7fd964009b30).stop 2024-02-19T22:16:17.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.722+0000 7fd97a1fa700 1 -- 172.21.15.184:0/605397361 shutdown_connections 2024-02-19T22:16:17.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.722+0000 7fd97a1fa700 1 --2- 172.21.15.184:0/605397361 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9740f4740 0x7fd9740f4b60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:17.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.722+0000 7fd97a1fa700 1 -- 172.21.15.184:0/605397361 >> 172.21.15.184:0/605397361 conn(0x7fd9740f0260 msgr2=0x7fd9740f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:17.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.722+0000 7fd97a1fa700 1 -- 172.21.15.184:0/605397361 shutdown_connections 2024-02-19T22:16:17.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.722+0000 7fd97a1fa700 1 -- 172.21.15.184:0/605397361 wait complete. 2024-02-19T22:16:17.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.722+0000 7fd97a1fa700 1 Processor -- start 2024-02-19T22:16:17.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.723+0000 7fd97a1fa700 1 -- start start 2024-02-19T22:16:17.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.723+0000 7fd97a1fa700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9740f4740 0x7fd9741067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:17.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.723+0000 7fd97a1fa700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd974106ca0 con 0x7fd9740f4740 2024-02-19T22:16:17.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.723+0000 7fd9737fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9740f4740 0x7fd9741067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:17.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.723+0000 7fd9737fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9740f4740 0x7fd9741067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:43560/0 (socket says 172.21.15.184:43560) 2024-02-19T22:16:17.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.723+0000 7fd9737fe700 1 -- 172.21.15.184:0/2558852737 learned_addr learned my addr 172.21.15.184:0/2558852737 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:17.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.723+0000 7fd9737fe700 1 -- 172.21.15.184:0/2558852737 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd964005040 con 0x7fd9740f4740 2024-02-19T22:16:17.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.723+0000 7fd9737fe700 1 --2- 172.21.15.184:0/2558852737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9740f4740 0x7fd9741067b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fd964005010 tx=0x7fd96400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:17.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.723+0000 7fd95bfff700 1 -- 172.21.15.184:0/2558852737 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd964009320 con 0x7fd9740f4740 2024-02-19T22:16:17.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.723+0000 7fd97a1fa700 1 -- 172.21.15.184:0/2558852737 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd974106ea0 con 0x7fd9740f4740 2024-02-19T22:16:17.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.723+0000 7fd95bfff700 1 -- 172.21.15.184:0/2558852737 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd964013380 con 0x7fd9740f4740 2024-02-19T22:16:17.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.723+0000 7fd95bfff700 1 -- 172.21.15.184:0/2558852737 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd964014e50 con 0x7fd9740f4740 2024-02-19T22:16:17.726 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.723+0000 7fd97a1fa700 1 -- 172.21.15.184:0/2558852737 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd974107330 con 0x7fd9740f4740 2024-02-19T22:16:17.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.724+0000 7fd95bfff700 1 -- 172.21.15.184:0/2558852737 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9640095f0 con 0x7fd9740f4740 2024-02-19T22:16:17.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.724+0000 7fd97a1fa700 1 -- 172.21.15.184:0/2558852737 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd974045ad0 con 0x7fd9740f4740 2024-02-19T22:16:17.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.724+0000 7fd95bfff700 1 --2- 172.21.15.184:0/2558852737 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd95c02ff10 0x7fd95c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:17.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.724+0000 7fd95bfff700 1 -- 172.21.15.184:0/2558852737 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd964040f80 con 0x7fd9740f4740 2024-02-19T22:16:17.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.725+0000 7fd972ffd700 1 --2- 172.21.15.184:0/2558852737 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd95c02ff10 0x7fd95c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:17.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.725+0000 7fd972ffd700 1 --2- 172.21.15.184:0/2558852737 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd95c02ff10 0x7fd95c0323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fd9680096a0 tx=0x7fd968006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:17.730 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.729+0000 7fd95bfff700 1 -- 172.21.15.184:0/2558852737 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd964014420 con 0x7fd9740f4740 2024-02-19T22:16:17.938 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.936+0000 7fd97a1fa700 1 -- 172.21.15.184:0/2558852737 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9741081e0 con 0x7fd9740f4740 2024-02-19T22:16:17.939 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.937+0000 7fd95bfff700 1 -- 172.21.15.184:0/2558852737 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd964010230 con 0x7fd9740f4740 2024-02-19T22:16:17.939 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:17.939 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:17.940 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.938+0000 7fd97a1fa700 1 -- 172.21.15.184:0/2558852737 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd95c02ff10 msgr2=0x7fd95c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:17.940 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.938+0000 7fd97a1fa700 1 --2- 172.21.15.184:0/2558852737 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd95c02ff10 0x7fd95c0323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fd9680096a0 tx=0x7fd968006b40).stop 2024-02-19T22:16:17.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.938+0000 7fd97a1fa700 1 -- 172.21.15.184:0/2558852737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9740f4740 msgr2=0x7fd9741067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:17.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.938+0000 7fd97a1fa700 1 --2- 172.21.15.184:0/2558852737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9740f4740 0x7fd9741067b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fd964005010 tx=0x7fd96400b040).stop 2024-02-19T22:16:17.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.939+0000 7fd97a1fa700 1 -- 172.21.15.184:0/2558852737 shutdown_connections 2024-02-19T22:16:17.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.939+0000 7fd97a1fa700 1 --2- 172.21.15.184:0/2558852737 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd95c02ff10 0x7fd95c0323d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:17.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.939+0000 7fd97a1fa700 1 --2- 172.21.15.184:0/2558852737 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9740f4740 0x7fd9741067b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:17.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.939+0000 7fd97a1fa700 1 -- 172.21.15.184:0/2558852737 >> 172.21.15.184:0/2558852737 conn(0x7fd9740f0260 msgr2=0x7fd9740f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:17.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.939+0000 7fd97a1fa700 1 -- 172.21.15.184:0/2558852737 shutdown_connections 2024-02-19T22:16:17.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:17.939+0000 7fd97a1fa700 1 -- 172.21.15.184:0/2558852737 wait complete. 2024-02-19T22:16:17.941 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:18.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:17 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:15.900952+0000 mgr.smithi176.faoogw (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:19.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:18 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:17.938227+0000 mon.smithi176 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.184:0/2558852737' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:19.619 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:19.619 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:20.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:17.901311+0000 mgr.smithi176.faoogw (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:21.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.547+0000 7f87e6043700 1 -- 172.21.15.184:0/1162417103 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87e00f62a0 msgr2=0x7f87e00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:21.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.547+0000 7f87e6043700 1 --2- 172.21.15.184:0/1162417103 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87e00f62a0 0x7f87e00f66c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f87dc009fb0 tx=0x7f87dc009b30).stop 2024-02-19T22:16:21.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.547+0000 7f87e6043700 1 -- 172.21.15.184:0/1162417103 shutdown_connections 2024-02-19T22:16:21.550 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.547+0000 7f87e6043700 1 --2- 172.21.15.184:0/1162417103 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87e00f62a0 0x7f87e00f66c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:21.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.547+0000 7f87e6043700 1 -- 172.21.15.184:0/1162417103 >> 172.21.15.184:0/1162417103 conn(0x7f87e00f1aa0 msgr2=0x7f87e00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:21.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.547+0000 7f87e6043700 1 -- 172.21.15.184:0/1162417103 shutdown_connections 2024-02-19T22:16:21.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.547+0000 7f87e6043700 1 -- 172.21.15.184:0/1162417103 wait complete. 2024-02-19T22:16:21.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.547+0000 7f87e6043700 1 Processor -- start 2024-02-19T22:16:21.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.547+0000 7f87e6043700 1 -- start start 2024-02-19T22:16:21.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.548+0000 7f87e6043700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87e00f62a0 0x7f87e01065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:21.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.548+0000 7f87e6043700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e0106ac0 con 0x7f87e00f62a0 2024-02-19T22:16:21.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.548+0000 7f87e5041700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87e00f62a0 0x7f87e01065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:21.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.548+0000 7f87e5041700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87e00f62a0 0x7f87e01065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:43586/0 (socket says 172.21.15.184:43586) 2024-02-19T22:16:21.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.548+0000 7f87e5041700 1 -- 172.21.15.184:0/1632072283 learned_addr learned my addr 172.21.15.184:0/1632072283 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:21.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.548+0000 7f87e5041700 1 -- 172.21.15.184:0/1632072283 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc005040 con 0x7f87e00f62a0 2024-02-19T22:16:21.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.548+0000 7f87e5041700 1 --2- 172.21.15.184:0/1632072283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87e00f62a0 0x7f87e01065d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f87dc0089f0 tx=0x7f87dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:21.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.548+0000 7f87d5ffb700 1 -- 172.21.15.184:0/1632072283 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87dc009320 con 0x7f87e00f62a0 2024-02-19T22:16:21.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.548+0000 7f87e6043700 1 -- 172.21.15.184:0/1632072283 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e0106cc0 con 0x7f87e00f62a0 2024-02-19T22:16:21.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.548+0000 7f87d5ffb700 1 -- 172.21.15.184:0/1632072283 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87dc013380 con 0x7f87e00f62a0 2024-02-19T22:16:21.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.548+0000 7f87d5ffb700 1 -- 172.21.15.184:0/1632072283 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87dc014e20 con 0x7f87e00f62a0 2024-02-19T22:16:21.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.549+0000 7f87e6043700 1 -- 172.21.15.184:0/1632072283 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e0107150 con 0x7f87e00f62a0 2024-02-19T22:16:21.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.549+0000 7f87d5ffb700 1 -- 172.21.15.184:0/1632072283 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87dc014420 con 0x7f87e00f62a0 2024-02-19T22:16:21.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.549+0000 7f87e6043700 1 -- 172.21.15.184:0/1632072283 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e0045ad0 con 0x7f87e00f62a0 2024-02-19T22:16:21.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.550+0000 7f87d5ffb700 1 --2- 172.21.15.184:0/1632072283 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f87cc02ff10 0x7f87cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:21.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.550+0000 7f87d5ffb700 1 -- 172.21.15.184:0/1632072283 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87dc040ef0 con 0x7f87e00f62a0 2024-02-19T22:16:21.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.550+0000 7f87e4840700 1 --2- 172.21.15.184:0/1632072283 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f87cc02ff10 0x7f87cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:21.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.550+0000 7f87e4840700 1 --2- 172.21.15.184:0/1632072283 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f87cc02ff10 0x7f87cc0323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f87d00096a0 tx=0x7f87d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:21.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.554+0000 7f87d5ffb700 1 -- 172.21.15.184:0/1632072283 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87dc014790 con 0x7f87e00f62a0 2024-02-19T22:16:21.760 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.759+0000 7f87e6043700 1 -- 172.21.15.184:0/1632072283 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87e0107fa0 con 0x7f87e00f62a0 2024-02-19T22:16:21.761 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.759+0000 7f87d5ffb700 1 -- 172.21.15.184:0/1632072283 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f87dc01d7f0 con 0x7f87e00f62a0 2024-02-19T22:16:21.761 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:21.761 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:21.763 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.761+0000 7f87e6043700 1 -- 172.21.15.184:0/1632072283 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f87cc02ff10 msgr2=0x7f87cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:21.763 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.761+0000 7f87e6043700 1 --2- 172.21.15.184:0/1632072283 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f87cc02ff10 0x7f87cc0323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f87d00096a0 tx=0x7f87d0006b40).stop 2024-02-19T22:16:21.763 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.761+0000 7f87e6043700 1 -- 172.21.15.184:0/1632072283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87e00f62a0 msgr2=0x7f87e01065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:21.763 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.761+0000 7f87e6043700 1 --2- 172.21.15.184:0/1632072283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87e00f62a0 0x7f87e01065d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f87dc0089f0 tx=0x7f87dc00b040).stop 2024-02-19T22:16:21.763 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.761+0000 7f87e6043700 1 -- 172.21.15.184:0/1632072283 shutdown_connections 2024-02-19T22:16:21.763 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.761+0000 7f87e6043700 1 --2- 172.21.15.184:0/1632072283 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f87cc02ff10 0x7f87cc0323d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:21.763 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.761+0000 7f87e6043700 1 --2- 172.21.15.184:0/1632072283 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f87e00f62a0 0x7f87e01065d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:21.763 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.761+0000 7f87e6043700 1 -- 172.21.15.184:0/1632072283 >> 172.21.15.184:0/1632072283 conn(0x7f87e00f1aa0 msgr2=0x7f87e00f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:21.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.761+0000 7f87e6043700 1 -- 172.21.15.184:0/1632072283 shutdown_connections 2024-02-19T22:16:21.764 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:21.761+0000 7f87e6043700 1 -- 172.21.15.184:0/1632072283 wait complete. 2024-02-19T22:16:21.764 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:22.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:19.901697+0000 mgr.smithi176.faoogw (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:22.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:21.760629+0000 mon.smithi176 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.184:0/1632072283' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:23.430 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:23.431 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:24.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:21.901998+0000 mgr.smithi176.faoogw (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:25.221 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.218+0000 7fe74175a700 1 -- 172.21.15.184:0/216911603 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe73c0f5410 msgr2=0x7fe73c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:25.221 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.218+0000 7fe74175a700 1 --2- 172.21.15.184:0/216911603 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe73c0f5410 0x7fe73c0f5830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fe730009fb0 tx=0x7fe730009b30).stop 2024-02-19T22:16:25.221 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.219+0000 7fe74175a700 1 -- 172.21.15.184:0/216911603 shutdown_connections 2024-02-19T22:16:25.222 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.219+0000 7fe74175a700 1 --2- 172.21.15.184:0/216911603 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe73c0f5410 0x7fe73c0f5830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:25.222 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.219+0000 7fe74175a700 1 -- 172.21.15.184:0/216911603 >> 172.21.15.184:0/216911603 conn(0x7fe73c0f0b70 msgr2=0x7fe73c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:25.222 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.219+0000 7fe74175a700 1 -- 172.21.15.184:0/216911603 shutdown_connections 2024-02-19T22:16:25.222 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.219+0000 7fe74175a700 1 -- 172.21.15.184:0/216911603 wait complete. 2024-02-19T22:16:25.222 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.219+0000 7fe74175a700 1 Processor -- start 2024-02-19T22:16:25.222 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.219+0000 7fe74175a700 1 -- start start 2024-02-19T22:16:25.222 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.220+0000 7fe74175a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe73c0f5410 0x7fe73c109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:25.222 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.220+0000 7fe74175a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73c060220 con 0x7fe73c0f5410 2024-02-19T22:16:25.223 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.220+0000 7fe73affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe73c0f5410 0x7fe73c109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:25.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.220+0000 7fe73affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe73c0f5410 0x7fe73c109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:43594/0 (socket says 172.21.15.184:43594) 2024-02-19T22:16:25.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.220+0000 7fe73affd700 1 -- 172.21.15.184:0/250402857 learned_addr learned my addr 172.21.15.184:0/250402857 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:25.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.220+0000 7fe73affd700 1 -- 172.21.15.184:0/250402857 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe730005040 con 0x7fe73c0f5410 2024-02-19T22:16:25.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.220+0000 7fe73affd700 1 --2- 172.21.15.184:0/250402857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe73c0f5410 0x7fe73c109540 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fe730005010 tx=0x7fe73000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:25.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.220+0000 7fe7237fe700 1 -- 172.21.15.184:0/250402857 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe730009320 con 0x7fe73c0f5410 2024-02-19T22:16:25.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.220+0000 7fe7237fe700 1 -- 172.21.15.184:0/250402857 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe730013380 con 0x7fe73c0f5410 2024-02-19T22:16:25.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.220+0000 7fe74175a700 1 -- 172.21.15.184:0/250402857 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe73c060420 con 0x7fe73c0f5410 2024-02-19T22:16:25.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.220+0000 7fe7237fe700 1 -- 172.21.15.184:0/250402857 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe730014e50 con 0x7fe73c0f5410 2024-02-19T22:16:25.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.220+0000 7fe74175a700 1 -- 172.21.15.184:0/250402857 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73c109bf0 con 0x7fe73c0f5410 2024-02-19T22:16:25.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.221+0000 7fe7237fe700 1 -- 172.21.15.184:0/250402857 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe730014420 con 0x7fe73c0f5410 2024-02-19T22:16:25.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.221+0000 7fe7237fe700 1 --2- 172.21.15.184:0/250402857 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe72402ff10 0x7fe7240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:25.224 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.221+0000 7fe74175a700 1 -- 172.21.15.184:0/250402857 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73c059100 con 0x7fe73c0f5410 2024-02-19T22:16:25.225 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.221+0000 7fe7237fe700 1 -- 172.21.15.184:0/250402857 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe730040f40 con 0x7fe73c0f5410 2024-02-19T22:16:25.225 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.222+0000 7fe73a7fc700 1 --2- 172.21.15.184:0/250402857 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe72402ff10 0x7fe7240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:25.225 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.222+0000 7fe73a7fc700 1 --2- 172.21.15.184:0/250402857 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe72402ff10 0x7fe7240323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fe72c0096a0 tx=0x7fe72c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:25.228 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.226+0000 7fe7237fe700 1 -- 172.21.15.184:0/250402857 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe73001db70 con 0x7fe73c0f5410 2024-02-19T22:16:25.432 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.430+0000 7fe74175a700 1 -- 172.21.15.184:0/250402857 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe73c0fafa0 con 0x7fe73c0f5410 2024-02-19T22:16:25.433 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.431+0000 7fe7237fe700 1 -- 172.21.15.184:0/250402857 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe73001db70 con 0x7fe73c0f5410 2024-02-19T22:16:25.433 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:25.433 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:25.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.432+0000 7fe74175a700 1 -- 172.21.15.184:0/250402857 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe72402ff10 msgr2=0x7fe7240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:25.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.432+0000 7fe74175a700 1 --2- 172.21.15.184:0/250402857 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe72402ff10 0x7fe7240323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fe72c0096a0 tx=0x7fe72c006b40).stop 2024-02-19T22:16:25.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.432+0000 7fe74175a700 1 -- 172.21.15.184:0/250402857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe73c0f5410 msgr2=0x7fe73c109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:25.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.432+0000 7fe74175a700 1 --2- 172.21.15.184:0/250402857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe73c0f5410 0x7fe73c109540 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fe730005010 tx=0x7fe73000b040).stop 2024-02-19T22:16:25.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.433+0000 7fe74175a700 1 -- 172.21.15.184:0/250402857 shutdown_connections 2024-02-19T22:16:25.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.433+0000 7fe74175a700 1 --2- 172.21.15.184:0/250402857 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe72402ff10 0x7fe7240323d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:25.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.433+0000 7fe74175a700 1 --2- 172.21.15.184:0/250402857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe73c0f5410 0x7fe73c109540 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:25.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.433+0000 7fe74175a700 1 -- 172.21.15.184:0/250402857 >> 172.21.15.184:0/250402857 conn(0x7fe73c0f0b70 msgr2=0x7fe73c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:25.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.433+0000 7fe74175a700 1 -- 172.21.15.184:0/250402857 shutdown_connections 2024-02-19T22:16:25.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:25.433+0000 7fe74175a700 1 -- 172.21.15.184:0/250402857 wait complete. 2024-02-19T22:16:25.436 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:26.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:23.902379+0000 mgr.smithi176.faoogw (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:26.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:25.432341+0000 mon.smithi176 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.184:0/250402857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:27.114 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:27.114 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:28.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:25.902735+0000 mgr.smithi176.faoogw (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:29.028 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.025+0000 7feed4a8b700 1 -- 172.21.15.184:0/1409525512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feed00f5410 msgr2=0x7feed00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:29.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.025+0000 7feed4a8b700 1 --2- 172.21.15.184:0/1409525512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feed00f5410 0x7feed00f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7feec4009fb0 tx=0x7feec4009b30).stop 2024-02-19T22:16:29.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.025+0000 7feed4a8b700 1 -- 172.21.15.184:0/1409525512 shutdown_connections 2024-02-19T22:16:29.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.025+0000 7feed4a8b700 1 --2- 172.21.15.184:0/1409525512 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feed00f5410 0x7feed00f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:29.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.025+0000 7feed4a8b700 1 -- 172.21.15.184:0/1409525512 >> 172.21.15.184:0/1409525512 conn(0x7feed00f0b70 msgr2=0x7feed00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:29.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.025+0000 7feed4a8b700 1 -- 172.21.15.184:0/1409525512 shutdown_connections 2024-02-19T22:16:29.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.025+0000 7feed4a8b700 1 -- 172.21.15.184:0/1409525512 wait complete. 2024-02-19T22:16:29.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.026+0000 7feed4a8b700 1 Processor -- start 2024-02-19T22:16:29.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.026+0000 7feed4a8b700 1 -- start start 2024-02-19T22:16:29.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.026+0000 7feed4a8b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feed00f5410 0x7feed01056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:29.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.026+0000 7feed4a8b700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed0105bd0 con 0x7feed00f5410 2024-02-19T22:16:29.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.026+0000 7feece59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feed00f5410 0x7feed01056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:29.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.026+0000 7feece59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feed00f5410 0x7feed01056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:42930/0 (socket says 172.21.15.184:42930) 2024-02-19T22:16:29.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.026+0000 7feece59c700 1 -- 172.21.15.184:0/1919012764 learned_addr learned my addr 172.21.15.184:0/1919012764 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:29.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.026+0000 7feece59c700 1 -- 172.21.15.184:0/1919012764 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec4005040 con 0x7feed00f5410 2024-02-19T22:16:29.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.026+0000 7feece59c700 1 --2- 172.21.15.184:0/1919012764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feed00f5410 0x7feed01056e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7feec4005010 tx=0x7feec400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:29.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.027+0000 7feebeffd700 1 -- 172.21.15.184:0/1919012764 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feec4009320 con 0x7feed00f5410 2024-02-19T22:16:29.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.027+0000 7feed4a8b700 1 -- 172.21.15.184:0/1919012764 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed0105dd0 con 0x7feed00f5410 2024-02-19T22:16:29.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.027+0000 7feebeffd700 1 -- 172.21.15.184:0/1919012764 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feec4013380 con 0x7feed00f5410 2024-02-19T22:16:29.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.027+0000 7feebeffd700 1 -- 172.21.15.184:0/1919012764 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feec4014e50 con 0x7feed00f5410 2024-02-19T22:16:29.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.027+0000 7feed4a8b700 1 -- 172.21.15.184:0/1919012764 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed0106260 con 0x7feed00f5410 2024-02-19T22:16:29.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.027+0000 7feebeffd700 1 -- 172.21.15.184:0/1919012764 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feec40095f0 con 0x7feed00f5410 2024-02-19T22:16:29.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.028+0000 7feebeffd700 1 --2- 172.21.15.184:0/1919012764 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7feeb802ff10 0x7feeb80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:29.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.028+0000 7feed4a8b700 1 -- 172.21.15.184:0/1919012764 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed0045ad0 con 0x7feed00f5410 2024-02-19T22:16:29.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.028+0000 7feebeffd700 1 -- 172.21.15.184:0/1919012764 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feec4040f80 con 0x7feed00f5410 2024-02-19T22:16:29.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.028+0000 7feecdd9b700 1 --2- 172.21.15.184:0/1919012764 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7feeb802ff10 0x7feeb80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:29.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.028+0000 7feecdd9b700 1 --2- 172.21.15.184:0/1919012764 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7feeb802ff10 0x7feeb80323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7feec00096a0 tx=0x7feec0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:29.034 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.032+0000 7feebeffd700 1 -- 172.21.15.184:0/1919012764 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feec4014420 con 0x7feed00f5410 2024-02-19T22:16:29.239 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.237+0000 7feed4a8b700 1 -- 172.21.15.184:0/1919012764 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feed0107110 con 0x7feed00f5410 2024-02-19T22:16:29.239 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.238+0000 7feebeffd700 1 -- 172.21.15.184:0/1919012764 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feec4010230 con 0x7feed00f5410 2024-02-19T22:16:29.240 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:29.240 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:29.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.239+0000 7feed4a8b700 1 -- 172.21.15.184:0/1919012764 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7feeb802ff10 msgr2=0x7feeb80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:29.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.239+0000 7feed4a8b700 1 --2- 172.21.15.184:0/1919012764 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7feeb802ff10 0x7feeb80323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7feec00096a0 tx=0x7feec0006b40).stop 2024-02-19T22:16:29.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.239+0000 7feed4a8b700 1 -- 172.21.15.184:0/1919012764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feed00f5410 msgr2=0x7feed01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:29.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.239+0000 7feed4a8b700 1 --2- 172.21.15.184:0/1919012764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feed00f5410 0x7feed01056e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7feec4005010 tx=0x7feec400b040).stop 2024-02-19T22:16:29.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.239+0000 7feed4a8b700 1 -- 172.21.15.184:0/1919012764 shutdown_connections 2024-02-19T22:16:29.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.240+0000 7feed4a8b700 1 --2- 172.21.15.184:0/1919012764 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7feeb802ff10 0x7feeb80323d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:29.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.240+0000 7feed4a8b700 1 --2- 172.21.15.184:0/1919012764 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feed00f5410 0x7feed01056e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:29.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.240+0000 7feed4a8b700 1 -- 172.21.15.184:0/1919012764 >> 172.21.15.184:0/1919012764 conn(0x7feed00f0b70 msgr2=0x7feed00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:29.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.240+0000 7feed4a8b700 1 -- 172.21.15.184:0/1919012764 shutdown_connections 2024-02-19T22:16:29.242 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:29.240+0000 7feed4a8b700 1 -- 172.21.15.184:0/1919012764 wait complete. 2024-02-19T22:16:29.242 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:30.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:29 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:27.903091+0000 mgr.smithi176.faoogw (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:30.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:29 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:29.239265+0000 mon.smithi176 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.184:0/1919012764' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:30.929 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:30.929 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:32.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:29.903449+0000 mgr.smithi176.faoogw (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:32.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.938+0000 7f8b465ad700 1 -- 172.21.15.184:0/3346966233 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b400f6510 msgr2=0x7f8b400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:32.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.938+0000 7f8b465ad700 1 --2- 172.21.15.184:0/3346966233 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b400f6510 0x7f8b400f6930 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f8b30009fb0 tx=0x7f8b30009b30).stop 2024-02-19T22:16:32.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.938+0000 7f8b465ad700 1 -- 172.21.15.184:0/3346966233 shutdown_connections 2024-02-19T22:16:32.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.938+0000 7f8b465ad700 1 --2- 172.21.15.184:0/3346966233 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b400f6510 0x7f8b400f6930 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:32.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.938+0000 7f8b465ad700 1 -- 172.21.15.184:0/3346966233 >> 172.21.15.184:0/3346966233 conn(0x7f8b400f1c70 msgr2=0x7f8b400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:32.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.938+0000 7f8b465ad700 1 -- 172.21.15.184:0/3346966233 shutdown_connections 2024-02-19T22:16:32.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.938+0000 7f8b465ad700 1 -- 172.21.15.184:0/3346966233 wait complete. 2024-02-19T22:16:32.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.939+0000 7f8b465ad700 1 Processor -- start 2024-02-19T22:16:32.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.939+0000 7f8b465ad700 1 -- start start 2024-02-19T22:16:32.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.939+0000 7f8b465ad700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b400f6510 0x7f8b4006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:32.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.939+0000 7f8b465ad700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b40069e20 con 0x7f8b400f6510 2024-02-19T22:16:32.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.940+0000 7f8b3ffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b400f6510 0x7f8b4006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:32.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.940+0000 7f8b3ffff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b400f6510 0x7f8b4006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:42946/0 (socket says 172.21.15.184:42946) 2024-02-19T22:16:32.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.940+0000 7f8b3ffff700 1 -- 172.21.15.184:0/1451089335 learned_addr learned my addr 172.21.15.184:0/1451089335 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:32.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.940+0000 7f8b3ffff700 1 -- 172.21.15.184:0/1451089335 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b30005040 con 0x7f8b400f6510 2024-02-19T22:16:32.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.940+0000 7f8b3ffff700 1 --2- 172.21.15.184:0/1451089335 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b400f6510 0x7f8b4006b9a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f8b30005010 tx=0x7f8b3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:32.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.940+0000 7f8b3cff9700 1 -- 172.21.15.184:0/1451089335 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b30009320 con 0x7f8b400f6510 2024-02-19T22:16:32.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.940+0000 7f8b465ad700 1 -- 172.21.15.184:0/1451089335 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b40069fc0 con 0x7f8b400f6510 2024-02-19T22:16:32.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.940+0000 7f8b3cff9700 1 -- 172.21.15.184:0/1451089335 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b30013380 con 0x7f8b400f6510 2024-02-19T22:16:32.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.940+0000 7f8b3cff9700 1 -- 172.21.15.184:0/1451089335 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b30014e50 con 0x7f8b400f6510 2024-02-19T22:16:32.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.940+0000 7f8b465ad700 1 -- 172.21.15.184:0/1451089335 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b4006a450 con 0x7f8b400f6510 2024-02-19T22:16:32.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.941+0000 7f8b3cff9700 1 -- 172.21.15.184:0/1451089335 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b300095f0 con 0x7f8b400f6510 2024-02-19T22:16:32.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.941+0000 7f8b465ad700 1 -- 172.21.15.184:0/1451089335 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b40059100 con 0x7f8b400f6510 2024-02-19T22:16:32.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.941+0000 7f8b3cff9700 1 --2- 172.21.15.184:0/1451089335 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8b2802ff10 0x7f8b280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:32.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.941+0000 7f8b3cff9700 1 -- 172.21.15.184:0/1451089335 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b30040e60 con 0x7f8b400f6510 2024-02-19T22:16:32.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.941+0000 7f8b3f7fe700 1 --2- 172.21.15.184:0/1451089335 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8b2802ff10 0x7f8b280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:32.945 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.942+0000 7f8b3f7fe700 1 --2- 172.21.15.184:0/1451089335 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8b2802ff10 0x7f8b280323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f8b340096a0 tx=0x7f8b34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:32.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:32.946+0000 7f8b3cff9700 1 -- 172.21.15.184:0/1451089335 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b30014420 con 0x7f8b400f6510 2024-02-19T22:16:33.152 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:33.150+0000 7f8b465ad700 1 -- 172.21.15.184:0/1451089335 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b400fc0a0 con 0x7f8b400f6510 2024-02-19T22:16:33.153 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:33.151+0000 7f8b3cff9700 1 -- 172.21.15.184:0/1451089335 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8b3001d980 con 0x7f8b400f6510 2024-02-19T22:16:33.153 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:33.153 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:33.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:33.153+0000 7f8b465ad700 1 -- 172.21.15.184:0/1451089335 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8b2802ff10 msgr2=0x7f8b280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:33.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:33.153+0000 7f8b465ad700 1 --2- 172.21.15.184:0/1451089335 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8b2802ff10 0x7f8b280323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f8b340096a0 tx=0x7f8b34006b40).stop 2024-02-19T22:16:33.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:33.153+0000 7f8b465ad700 1 -- 172.21.15.184:0/1451089335 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b400f6510 msgr2=0x7f8b4006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:33.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:33.153+0000 7f8b465ad700 1 --2- 172.21.15.184:0/1451089335 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b400f6510 0x7f8b4006b9a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f8b30005010 tx=0x7f8b3000b040).stop 2024-02-19T22:16:33.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:33.153+0000 7f8b465ad700 1 -- 172.21.15.184:0/1451089335 shutdown_connections 2024-02-19T22:16:33.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:33.153+0000 7f8b465ad700 1 --2- 172.21.15.184:0/1451089335 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8b2802ff10 0x7f8b280323d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:33.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:33.153+0000 7f8b465ad700 1 --2- 172.21.15.184:0/1451089335 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b400f6510 0x7f8b4006b9a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:33.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:33.153+0000 7f8b465ad700 1 -- 172.21.15.184:0/1451089335 >> 172.21.15.184:0/1451089335 conn(0x7f8b400f1c70 msgr2=0x7f8b400f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:33.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:33.153+0000 7f8b465ad700 1 -- 172.21.15.184:0/1451089335 shutdown_connections 2024-02-19T22:16:33.156 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:33.153+0000 7f8b465ad700 1 -- 172.21.15.184:0/1451089335 wait complete. 2024-02-19T22:16:33.156 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:34.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:31.903741+0000 mgr.smithi176.faoogw (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:34.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:33.152525+0000 mon.smithi176 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.184:0/1451089335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:34.795 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:34.796 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:36.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:33.904023+0000 mgr.smithi176.faoogw (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:36.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.647+0000 7fe61edcc700 1 -- 172.21.15.184:0/46171452 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe618060220 msgr2=0x7fe6180f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:36.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.647+0000 7fe61edcc700 1 --2- 172.21.15.184:0/46171452 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe618060220 0x7fe6180f9150 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fe60c009fb0 tx=0x7fe60c009b30).stop 2024-02-19T22:16:36.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.647+0000 7fe61edcc700 1 -- 172.21.15.184:0/46171452 shutdown_connections 2024-02-19T22:16:36.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.647+0000 7fe61edcc700 1 --2- 172.21.15.184:0/46171452 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe618060220 0x7fe6180f9150 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:36.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.647+0000 7fe61edcc700 1 -- 172.21.15.184:0/46171452 >> 172.21.15.184:0/46171452 conn(0x7fe6180f0b70 msgr2=0x7fe6180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:36.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.647+0000 7fe61edcc700 1 -- 172.21.15.184:0/46171452 shutdown_connections 2024-02-19T22:16:36.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.647+0000 7fe61edcc700 1 -- 172.21.15.184:0/46171452 wait complete. 2024-02-19T22:16:36.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.648+0000 7fe61edcc700 1 Processor -- start 2024-02-19T22:16:36.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.648+0000 7fe61edcc700 1 -- start start 2024-02-19T22:16:36.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.648+0000 7fe61edcc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe618060220 0x7fe6181067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:36.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.648+0000 7fe61edcc700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe618106cb0 con 0x7fe618060220 2024-02-19T22:16:36.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.649+0000 7fe61cb68700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe618060220 0x7fe6181067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:36.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.649+0000 7fe61cb68700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe618060220 0x7fe6181067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:51442/0 (socket says 172.21.15.184:51442) 2024-02-19T22:16:36.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.649+0000 7fe61cb68700 1 -- 172.21.15.184:0/2762089180 learned_addr learned my addr 172.21.15.184:0/2762089180 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:36.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.649+0000 7fe61cb68700 1 -- 172.21.15.184:0/2762089180 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c005040 con 0x7fe618060220 2024-02-19T22:16:36.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.649+0000 7fe61cb68700 1 --2- 172.21.15.184:0/2762089180 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe618060220 0x7fe6181067c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fe60c0089f0 tx=0x7fe60c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:36.653 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.649+0000 7fe6157fa700 1 -- 172.21.15.184:0/2762089180 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe60c009320 con 0x7fe618060220 2024-02-19T22:16:36.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.649+0000 7fe6157fa700 1 -- 172.21.15.184:0/2762089180 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe60c013380 con 0x7fe618060220 2024-02-19T22:16:36.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.649+0000 7fe61edcc700 1 -- 172.21.15.184:0/2762089180 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe618106eb0 con 0x7fe618060220 2024-02-19T22:16:36.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.649+0000 7fe6157fa700 1 -- 172.21.15.184:0/2762089180 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe60c014e20 con 0x7fe618060220 2024-02-19T22:16:36.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.649+0000 7fe61edcc700 1 -- 172.21.15.184:0/2762089180 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe618107340 con 0x7fe618060220 2024-02-19T22:16:36.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.650+0000 7fe6157fa700 1 -- 172.21.15.184:0/2762089180 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe60c0263a0 con 0x7fe618060220 2024-02-19T22:16:36.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.650+0000 7fe6157fa700 1 --2- 172.21.15.184:0/2762089180 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe60802ff60 0x7fe608032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:36.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.650+0000 7fe6157fa700 1 -- 172.21.15.184:0/2762089180 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe60c040f00 con 0x7fe618060220 2024-02-19T22:16:36.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.650+0000 7fe61edcc700 1 -- 172.21.15.184:0/2762089180 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe618045ad0 con 0x7fe618060220 2024-02-19T22:16:36.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.650+0000 7fe617fff700 1 --2- 172.21.15.184:0/2762089180 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe60802ff60 0x7fe608032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:36.654 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.651+0000 7fe617fff700 1 --2- 172.21.15.184:0/2762089180 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe60802ff60 0x7fe608032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fe6040096a0 tx=0x7fe604006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:36.656 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.655+0000 7fe6157fa700 1 -- 172.21.15.184:0/2762089180 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe60c012920 con 0x7fe618060220 2024-02-19T22:16:36.863 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.862+0000 7fe61edcc700 1 -- 172.21.15.184:0/2762089180 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe618108060 con 0x7fe618060220 2024-02-19T22:16:36.864 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.862+0000 7fe6157fa700 1 -- 172.21.15.184:0/2762089180 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe60c02b020 con 0x7fe618060220 2024-02-19T22:16:36.864 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:36.865 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:36.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.864+0000 7fe61edcc700 1 -- 172.21.15.184:0/2762089180 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe60802ff60 msgr2=0x7fe608032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:36.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.864+0000 7fe61edcc700 1 --2- 172.21.15.184:0/2762089180 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe60802ff60 0x7fe608032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fe6040096a0 tx=0x7fe604006b40).stop 2024-02-19T22:16:36.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.864+0000 7fe61edcc700 1 -- 172.21.15.184:0/2762089180 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe618060220 msgr2=0x7fe6181067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:36.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.864+0000 7fe61edcc700 1 --2- 172.21.15.184:0/2762089180 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe618060220 0x7fe6181067c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fe60c0089f0 tx=0x7fe60c00b040).stop 2024-02-19T22:16:36.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.864+0000 7fe61edcc700 1 -- 172.21.15.184:0/2762089180 shutdown_connections 2024-02-19T22:16:36.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.864+0000 7fe61edcc700 1 --2- 172.21.15.184:0/2762089180 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe60802ff60 0x7fe608032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:36.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.864+0000 7fe61edcc700 1 --2- 172.21.15.184:0/2762089180 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe618060220 0x7fe6181067c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:36.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.864+0000 7fe61edcc700 1 -- 172.21.15.184:0/2762089180 >> 172.21.15.184:0/2762089180 conn(0x7fe6180f0b70 msgr2=0x7fe6180f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:36.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.864+0000 7fe61edcc700 1 -- 172.21.15.184:0/2762089180 shutdown_connections 2024-02-19T22:16:36.866 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:36.864+0000 7fe61edcc700 1 -- 172.21.15.184:0/2762089180 wait complete. 2024-02-19T22:16:36.867 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:37.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:35.973809+0000 mon.smithi176 (mon.0) 258 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:16:37.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:35.975639+0000 mon.smithi176 (mon.0) 259 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:16:38.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:35.904367+0000 mgr.smithi176.faoogw (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:38.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:36.863725+0000 mon.smithi176 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.184:0/2762089180' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:38.536 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:38.536 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:39.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:38 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:37.904683+0000 mgr.smithi176.faoogw (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:40.434 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.404+0000 7f27009ae700 1 -- 172.21.15.184:0/950998138 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0f5410 msgr2=0x7f26fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:40.434 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.404+0000 7f27009ae700 1 --2- 172.21.15.184:0/950998138 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0f5410 0x7f26fc0f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f26e4009fb0 tx=0x7f26e4009b30).stop 2024-02-19T22:16:40.434 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.404+0000 7f27009ae700 1 -- 172.21.15.184:0/950998138 shutdown_connections 2024-02-19T22:16:40.434 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.404+0000 7f27009ae700 1 --2- 172.21.15.184:0/950998138 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0f5410 0x7f26fc0f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:40.434 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.404+0000 7f27009ae700 1 -- 172.21.15.184:0/950998138 >> 172.21.15.184:0/950998138 conn(0x7f26fc0f0b70 msgr2=0x7f26fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:40.434 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.404+0000 7f27009ae700 1 -- 172.21.15.184:0/950998138 shutdown_connections 2024-02-19T22:16:40.434 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.404+0000 7f27009ae700 1 -- 172.21.15.184:0/950998138 wait complete. 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.404+0000 7f27009ae700 1 Processor -- start 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.404+0000 7f27009ae700 1 -- start start 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.404+0000 7f27009ae700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0f5410 0x7f26fc109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.404+0000 7f27009ae700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc060220 con 0x7f26fc0f5410 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.405+0000 7f26fa59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0f5410 0x7f26fc109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.405+0000 7f26fa59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0f5410 0x7f26fc109530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:51460/0 (socket says 172.21.15.184:51460) 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.405+0000 7f26fa59c700 1 -- 172.21.15.184:0/4062807493 learned_addr learned my addr 172.21.15.184:0/4062807493 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.405+0000 7f26fa59c700 1 -- 172.21.15.184:0/4062807493 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e4005040 con 0x7f26fc0f5410 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.405+0000 7f26fa59c700 1 --2- 172.21.15.184:0/4062807493 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0f5410 0x7f26fc109530 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f26e40089f0 tx=0x7f26e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.405+0000 7f26f2ffd700 1 -- 172.21.15.184:0/4062807493 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26e4009320 con 0x7f26fc0f5410 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.405+0000 7f27009ae700 1 -- 172.21.15.184:0/4062807493 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc060420 con 0x7f26fc0f5410 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.405+0000 7f26f2ffd700 1 -- 172.21.15.184:0/4062807493 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26e4013380 con 0x7f26fc0f5410 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.405+0000 7f26f2ffd700 1 -- 172.21.15.184:0/4062807493 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26e401ed80 con 0x7f26fc0f5410 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.405+0000 7f27009ae700 1 -- 172.21.15.184:0/4062807493 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26fc109be0 con 0x7f26fc0f5410 2024-02-19T22:16:40.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.406+0000 7f26f2ffd700 1 -- 172.21.15.184:0/4062807493 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26e401e420 con 0x7f26fc0f5410 2024-02-19T22:16:40.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.406+0000 7f27009ae700 1 -- 172.21.15.184:0/4062807493 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26fc059100 con 0x7f26fc0f5410 2024-02-19T22:16:40.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.406+0000 7f26f2ffd700 1 --2- 172.21.15.184:0/4062807493 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f26e802ff60 0x7f26e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:40.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.406+0000 7f26f2ffd700 1 -- 172.21.15.184:0/4062807493 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26e4041060 con 0x7f26fc0f5410 2024-02-19T22:16:40.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.407+0000 7f26f9d9b700 1 --2- 172.21.15.184:0/4062807493 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f26e802ff60 0x7f26e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:40.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.407+0000 7f26f9d9b700 1 --2- 172.21.15.184:0/4062807493 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f26e802ff60 0x7f26e8032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f26ec0096a0 tx=0x7f26ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:40.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.411+0000 7f26f2ffd700 1 -- 172.21.15.184:0/4062807493 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26e401cb10 con 0x7f26fc0f5410 2024-02-19T22:16:40.618 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.616+0000 7f27009ae700 1 -- 172.21.15.184:0/4062807493 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26fc045ad0 con 0x7f26fc0f5410 2024-02-19T22:16:40.619 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.617+0000 7f26f2ffd700 1 -- 172.21.15.184:0/4062807493 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f26e402c2d0 con 0x7f26fc0f5410 2024-02-19T22:16:40.619 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:40.619 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:40.621 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.619+0000 7f27009ae700 1 -- 172.21.15.184:0/4062807493 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f26e802ff60 msgr2=0x7f26e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:40.621 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.619+0000 7f27009ae700 1 --2- 172.21.15.184:0/4062807493 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f26e802ff60 0x7f26e8032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f26ec0096a0 tx=0x7f26ec006b40).stop 2024-02-19T22:16:40.621 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.619+0000 7f27009ae700 1 -- 172.21.15.184:0/4062807493 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0f5410 msgr2=0x7f26fc109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:40.621 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.619+0000 7f27009ae700 1 --2- 172.21.15.184:0/4062807493 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0f5410 0x7f26fc109530 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f26e40089f0 tx=0x7f26e400b040).stop 2024-02-19T22:16:40.621 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.619+0000 7f27009ae700 1 -- 172.21.15.184:0/4062807493 shutdown_connections 2024-02-19T22:16:40.621 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.619+0000 7f27009ae700 1 --2- 172.21.15.184:0/4062807493 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f26e802ff60 0x7f26e8032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:40.621 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.619+0000 7f27009ae700 1 --2- 172.21.15.184:0/4062807493 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f26fc0f5410 0x7f26fc109530 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:40.621 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.619+0000 7f27009ae700 1 -- 172.21.15.184:0/4062807493 >> 172.21.15.184:0/4062807493 conn(0x7f26fc0f0b70 msgr2=0x7f26fc0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:40.621 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.619+0000 7f27009ae700 1 -- 172.21.15.184:0/4062807493 shutdown_connections 2024-02-19T22:16:40.621 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:40.619+0000 7f27009ae700 1 -- 172.21.15.184:0/4062807493 wait complete. 2024-02-19T22:16:40.622 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:41.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:40 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:40.618470+0000 mon.smithi176 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.184:0/4062807493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:42.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:39.904941+0000 mgr.smithi176.faoogw (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:42.336 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:42.336 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:44.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:41.905258+0000 mgr.smithi176.faoogw (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:44.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.247+0000 7fa00b59e700 1 -- 172.21.15.184:0/3302607029 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c0f51a0 msgr2=0x7fa00c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:44.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.247+0000 7fa00b59e700 1 --2- 172.21.15.184:0/3302607029 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c0f51a0 0x7fa00c0f55c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f9ffc009fb0 tx=0x7f9ffc009b30).stop 2024-02-19T22:16:44.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.247+0000 7fa00b59e700 1 -- 172.21.15.184:0/3302607029 shutdown_connections 2024-02-19T22:16:44.250 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.247+0000 7fa00b59e700 1 --2- 172.21.15.184:0/3302607029 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c0f51a0 0x7fa00c0f55c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:44.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.247+0000 7fa00b59e700 1 -- 172.21.15.184:0/3302607029 >> 172.21.15.184:0/3302607029 conn(0x7fa00c0f09a0 msgr2=0x7fa00c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:44.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.247+0000 7fa00b59e700 1 -- 172.21.15.184:0/3302607029 shutdown_connections 2024-02-19T22:16:44.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.247+0000 7fa00b59e700 1 -- 172.21.15.184:0/3302607029 wait complete. 2024-02-19T22:16:44.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.247+0000 7fa00b59e700 1 Processor -- start 2024-02-19T22:16:44.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.247+0000 7fa00b59e700 1 -- start start 2024-02-19T22:16:44.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.248+0000 7fa00b59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c0f51a0 0x7fa00c1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:44.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.248+0000 7fa00b59e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00c1059c0 con 0x7fa00c0f51a0 2024-02-19T22:16:44.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.248+0000 7fa00a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c0f51a0 0x7fa00c1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:44.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.248+0000 7fa00a59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c0f51a0 0x7fa00c1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:51478/0 (socket says 172.21.15.184:51478) 2024-02-19T22:16:44.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.248+0000 7fa00a59c700 1 -- 172.21.15.184:0/1936468508 learned_addr learned my addr 172.21.15.184:0/1936468508 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:44.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.248+0000 7fa00a59c700 1 -- 172.21.15.184:0/1936468508 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ffc005040 con 0x7fa00c0f51a0 2024-02-19T22:16:44.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.248+0000 7fa00a59c700 1 --2- 172.21.15.184:0/1936468508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c0f51a0 0x7fa00c1054d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f9ffc0089f0 tx=0x7f9ffc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:44.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.248+0000 7f9ffaffd700 1 -- 172.21.15.184:0/1936468508 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ffc009320 con 0x7fa00c0f51a0 2024-02-19T22:16:44.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.248+0000 7f9ffaffd700 1 -- 172.21.15.184:0/1936468508 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ffc013380 con 0x7fa00c0f51a0 2024-02-19T22:16:44.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.248+0000 7fa00b59e700 1 -- 172.21.15.184:0/1936468508 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa00c105bc0 con 0x7fa00c0f51a0 2024-02-19T22:16:44.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.248+0000 7f9ffaffd700 1 -- 172.21.15.184:0/1936468508 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ffc014e20 con 0x7fa00c0f51a0 2024-02-19T22:16:44.252 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.248+0000 7fa00b59e700 1 -- 172.21.15.184:0/1936468508 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa00c106050 con 0x7fa00c0f51a0 2024-02-19T22:16:44.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.249+0000 7f9ffaffd700 1 -- 172.21.15.184:0/1936468508 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ffc0095f0 con 0x7fa00c0f51a0 2024-02-19T22:16:44.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.249+0000 7fa00b59e700 1 -- 172.21.15.184:0/1936468508 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa00c045ad0 con 0x7fa00c0f51a0 2024-02-19T22:16:44.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.249+0000 7f9ffaffd700 1 --2- 172.21.15.184:0/1936468508 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9ff402ff60 0x7f9ff4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:44.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.249+0000 7f9ffaffd700 1 -- 172.21.15.184:0/1936468508 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ffc040e40 con 0x7fa00c0f51a0 2024-02-19T22:16:44.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.250+0000 7fa009d9b700 1 --2- 172.21.15.184:0/1936468508 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9ff402ff60 0x7f9ff4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:44.253 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.250+0000 7fa009d9b700 1 --2- 172.21.15.184:0/1936468508 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9ff402ff60 0x7f9ff4032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fa0000096a0 tx=0x7fa000006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:44.256 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.254+0000 7f9ffaffd700 1 -- 172.21.15.184:0/1936468508 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ffc014420 con 0x7fa00c0f51a0 2024-02-19T22:16:44.462 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.460+0000 7fa00b59e700 1 -- 172.21.15.184:0/1936468508 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa00c106f00 con 0x7fa00c0f51a0 2024-02-19T22:16:44.463 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.461+0000 7f9ffaffd700 1 -- 172.21.15.184:0/1936468508 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9ffc01d980 con 0x7fa00c0f51a0 2024-02-19T22:16:44.463 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:44.463 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:44.464 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.462+0000 7fa00b59e700 1 -- 172.21.15.184:0/1936468508 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9ff402ff60 msgr2=0x7f9ff4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:44.465 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.462+0000 7fa00b59e700 1 --2- 172.21.15.184:0/1936468508 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9ff402ff60 0x7f9ff4032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fa0000096a0 tx=0x7fa000006b40).stop 2024-02-19T22:16:44.465 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.462+0000 7fa00b59e700 1 -- 172.21.15.184:0/1936468508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c0f51a0 msgr2=0x7fa00c1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:44.465 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.462+0000 7fa00b59e700 1 --2- 172.21.15.184:0/1936468508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c0f51a0 0x7fa00c1054d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f9ffc0089f0 tx=0x7f9ffc00b040).stop 2024-02-19T22:16:44.465 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.463+0000 7fa00b59e700 1 -- 172.21.15.184:0/1936468508 shutdown_connections 2024-02-19T22:16:44.465 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.463+0000 7fa00b59e700 1 --2- 172.21.15.184:0/1936468508 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9ff402ff60 0x7f9ff4032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:44.465 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.463+0000 7fa00b59e700 1 --2- 172.21.15.184:0/1936468508 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa00c0f51a0 0x7fa00c1054d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:44.465 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.463+0000 7fa00b59e700 1 -- 172.21.15.184:0/1936468508 >> 172.21.15.184:0/1936468508 conn(0x7fa00c0f09a0 msgr2=0x7fa00c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:44.465 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.463+0000 7fa00b59e700 1 -- 172.21.15.184:0/1936468508 shutdown_connections 2024-02-19T22:16:44.465 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:44.463+0000 7fa00b59e700 1 -- 172.21.15.184:0/1936468508 wait complete. 2024-02-19T22:16:44.465 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:45.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:44 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:44.462351+0000 mon.smithi176 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.184:0/1936468508' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:46.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:46.091 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:46.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:43.905607+0000 mgr.smithi176.faoogw (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:48.028 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.024+0000 7f9b4bfff700 1 -- 172.21.15.184:0/3991216724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b4c0f51a0 msgr2=0x7f9b4c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:48.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.024+0000 7f9b4bfff700 1 --2- 172.21.15.184:0/3991216724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b4c0f51a0 0x7f9b4c0f55c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f9b3c009fb0 tx=0x7f9b3c009b30).stop 2024-02-19T22:16:48.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.024+0000 7f9b4bfff700 1 -- 172.21.15.184:0/3991216724 shutdown_connections 2024-02-19T22:16:48.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.024+0000 7f9b4bfff700 1 --2- 172.21.15.184:0/3991216724 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b4c0f51a0 0x7f9b4c0f55c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:48.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.024+0000 7f9b4bfff700 1 -- 172.21.15.184:0/3991216724 >> 172.21.15.184:0/3991216724 conn(0x7f9b4c0f09a0 msgr2=0x7f9b4c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:48.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.024+0000 7f9b4bfff700 1 -- 172.21.15.184:0/3991216724 shutdown_connections 2024-02-19T22:16:48.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.025+0000 7f9b4bfff700 1 -- 172.21.15.184:0/3991216724 wait complete. 2024-02-19T22:16:48.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.025+0000 7f9b4bfff700 1 Processor -- start 2024-02-19T22:16:48.029 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.025+0000 7f9b4bfff700 1 -- start start 2024-02-19T22:16:48.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.025+0000 7f9b4bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b4c0f51a0 0x7f9b4c1087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:48.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.025+0000 7f9b4bfff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b4c108cc0 con 0x7f9b4c0f51a0 2024-02-19T22:16:48.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.025+0000 7f9b4affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b4c0f51a0 0x7f9b4c1087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:48.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.025+0000 7f9b4affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b4c0f51a0 0x7f9b4c1087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:41486/0 (socket says 172.21.15.184:41486) 2024-02-19T22:16:48.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.025+0000 7f9b4affd700 1 -- 172.21.15.184:0/2908004622 learned_addr learned my addr 172.21.15.184:0/2908004622 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:48.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.026+0000 7f9b4affd700 1 -- 172.21.15.184:0/2908004622 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c005040 con 0x7f9b4c0f51a0 2024-02-19T22:16:48.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.026+0000 7f9b4affd700 1 --2- 172.21.15.184:0/2908004622 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b4c0f51a0 0x7f9b4c1087d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f9b3c0089f0 tx=0x7f9b3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:48.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.026+0000 7f9b337fe700 1 -- 172.21.15.184:0/2908004622 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b3c009320 con 0x7f9b4c0f51a0 2024-02-19T22:16:48.030 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.026+0000 7f9b4bfff700 1 -- 172.21.15.184:0/2908004622 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b4c108ec0 con 0x7f9b4c0f51a0 2024-02-19T22:16:48.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.026+0000 7f9b337fe700 1 -- 172.21.15.184:0/2908004622 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b3c013380 con 0x7f9b4c0f51a0 2024-02-19T22:16:48.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.026+0000 7f9b337fe700 1 -- 172.21.15.184:0/2908004622 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b3c014e20 con 0x7f9b4c0f51a0 2024-02-19T22:16:48.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.026+0000 7f9b4bfff700 1 -- 172.21.15.184:0/2908004622 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b4c109350 con 0x7f9b4c0f51a0 2024-02-19T22:16:48.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.027+0000 7f9b337fe700 1 -- 172.21.15.184:0/2908004622 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b3c014420 con 0x7f9b4c0f51a0 2024-02-19T22:16:48.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.027+0000 7f9b4bfff700 1 -- 172.21.15.184:0/2908004622 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b4c045ad0 con 0x7f9b4c0f51a0 2024-02-19T22:16:48.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.027+0000 7f9b337fe700 1 --2- 172.21.15.184:0/2908004622 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9b3402ff10 0x7f9b340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:48.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.027+0000 7f9b337fe700 1 -- 172.21.15.184:0/2908004622 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b3c040ff0 con 0x7f9b4c0f51a0 2024-02-19T22:16:48.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.027+0000 7f9b4a7fc700 1 --2- 172.21.15.184:0/2908004622 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9b3402ff10 0x7f9b340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:48.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.028+0000 7f9b4a7fc700 1 --2- 172.21.15.184:0/2908004622 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9b3402ff10 0x7f9b340323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f9b400096a0 tx=0x7f9b40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:48.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.032+0000 7f9b337fe700 1 -- 172.21.15.184:0/2908004622 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b3c0095f0 con 0x7f9b4c0f51a0 2024-02-19T22:16:48.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:45.905877+0000 mgr.smithi176.faoogw (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:48.238 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.237+0000 7f9b4bfff700 1 -- 172.21.15.184:0/2908004622 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b4c10a200 con 0x7f9b4c0f51a0 2024-02-19T22:16:48.238 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.238+0000 7f9b337fe700 1 -- 172.21.15.184:0/2908004622 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9b3c01db70 con 0x7f9b4c0f51a0 2024-02-19T22:16:48.239 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:48.239 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:48.240 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.239+0000 7f9b4bfff700 1 -- 172.21.15.184:0/2908004622 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9b3402ff10 msgr2=0x7f9b340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:48.240 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.239+0000 7f9b4bfff700 1 --2- 172.21.15.184:0/2908004622 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9b3402ff10 0x7f9b340323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f9b400096a0 tx=0x7f9b40006b40).stop 2024-02-19T22:16:48.240 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.239+0000 7f9b4bfff700 1 -- 172.21.15.184:0/2908004622 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b4c0f51a0 msgr2=0x7f9b4c1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:48.240 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.239+0000 7f9b4bfff700 1 --2- 172.21.15.184:0/2908004622 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b4c0f51a0 0x7f9b4c1087d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f9b3c0089f0 tx=0x7f9b3c00b040).stop 2024-02-19T22:16:48.241 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.239+0000 7f9b4bfff700 1 -- 172.21.15.184:0/2908004622 shutdown_connections 2024-02-19T22:16:48.241 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.239+0000 7f9b4bfff700 1 --2- 172.21.15.184:0/2908004622 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f9b3402ff10 0x7f9b340323d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:48.241 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.239+0000 7f9b4bfff700 1 --2- 172.21.15.184:0/2908004622 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9b4c0f51a0 0x7f9b4c1087d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:48.241 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.239+0000 7f9b4bfff700 1 -- 172.21.15.184:0/2908004622 >> 172.21.15.184:0/2908004622 conn(0x7f9b4c0f09a0 msgr2=0x7f9b4c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:48.241 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.239+0000 7f9b4bfff700 1 -- 172.21.15.184:0/2908004622 shutdown_connections 2024-02-19T22:16:48.241 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:48.239+0000 7f9b4bfff700 1 -- 172.21.15.184:0/2908004622 wait complete. 2024-02-19T22:16:48.241 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:49.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:48 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:48.237950+0000 mon.smithi176 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.184:0/2908004622' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:49.963 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:49.964 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:50.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:49 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:47.906205+0000 mgr.smithi176.faoogw (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:52.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.028+0000 7fbc93aec700 1 -- 172.21.15.184:0/2887400046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc8c0f6510 msgr2=0x7fbc8c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:52.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.028+0000 7fbc93aec700 1 --2- 172.21.15.184:0/2887400046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c0f6930 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fbc7c009fb0 tx=0x7fbc7c009b30).stop 2024-02-19T22:16:52.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.028+0000 7fbc93aec700 1 -- 172.21.15.184:0/2887400046 shutdown_connections 2024-02-19T22:16:52.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.028+0000 7fbc93aec700 1 --2- 172.21.15.184:0/2887400046 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c0f6930 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:52.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.029+0000 7fbc93aec700 1 -- 172.21.15.184:0/2887400046 >> 172.21.15.184:0/2887400046 conn(0x7fbc8c0f1c70 msgr2=0x7fbc8c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:52.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.029+0000 7fbc93aec700 1 -- 172.21.15.184:0/2887400046 shutdown_connections 2024-02-19T22:16:52.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.029+0000 7fbc93aec700 1 -- 172.21.15.184:0/2887400046 wait complete. 2024-02-19T22:16:52.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.029+0000 7fbc93aec700 1 Processor -- start 2024-02-19T22:16:52.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.029+0000 7fbc93aec700 1 -- start start 2024-02-19T22:16:52.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.029+0000 7fbc93aec700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:52.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.029+0000 7fbc93aec700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c106cd0 con 0x7fbc8c0f6510 2024-02-19T22:16:52.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.029+0000 7fbc91888700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:52.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.029+0000 7fbc91888700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:41514/0 (socket says 172.21.15.184:41514) 2024-02-19T22:16:52.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.029+0000 7fbc91888700 1 -- 172.21.15.184:0/1615176618 learned_addr learned my addr 172.21.15.184:0/1615176618 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:52.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.030+0000 7fbc91888700 1 -- 172.21.15.184:0/1615176618 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c005040 con 0x7fbc8c0f6510 2024-02-19T22:16:52.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.030+0000 7fbc91888700 1 --2- 172.21.15.184:0/1615176618 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c1067e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fbc7c005010 tx=0x7fbc7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:52.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.030+0000 7fbc827fc700 1 -- 172.21.15.184:0/1615176618 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc7c009320 con 0x7fbc8c0f6510 2024-02-19T22:16:52.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.030+0000 7fbc93aec700 1 -- 172.21.15.184:0/1615176618 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8c106ed0 con 0x7fbc8c0f6510 2024-02-19T22:16:52.034 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.030+0000 7fbc827fc700 1 -- 172.21.15.184:0/1615176618 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc7c013380 con 0x7fbc8c0f6510 2024-02-19T22:16:52.034 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.030+0000 7fbc827fc700 1 -- 172.21.15.184:0/1615176618 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc7c014e50 con 0x7fbc8c0f6510 2024-02-19T22:16:52.034 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.030+0000 7fbc93aec700 1 -- 172.21.15.184:0/1615176618 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8c107360 con 0x7fbc8c0f6510 2024-02-19T22:16:52.034 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.031+0000 7fbc827fc700 1 -- 172.21.15.184:0/1615176618 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc7c014420 con 0x7fbc8c0f6510 2024-02-19T22:16:52.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.031+0000 7fbc93aec700 1 -- 172.21.15.184:0/1615176618 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8c045ad0 con 0x7fbc8c0f6510 2024-02-19T22:16:52.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.031+0000 7fbc827fc700 1 --2- 172.21.15.184:0/1615176618 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbc7802ff10 0x7fbc780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:52.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.031+0000 7fbc827fc700 1 -- 172.21.15.184:0/1615176618 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc7c040eb0 con 0x7fbc8c0f6510 2024-02-19T22:16:52.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.031+0000 7fbc91087700 1 --2- 172.21.15.184:0/1615176618 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbc7802ff10 0x7fbc780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:52.035 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.032+0000 7fbc91087700 1 --2- 172.21.15.184:0/1615176618 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbc7802ff10 0x7fbc780323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fbc880096a0 tx=0x7fbc88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:52.037 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.035+0000 7fbc827fc700 1 -- 172.21.15.184:0/1615176618 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc7c0095f0 con 0x7fbc8c0f6510 2024-02-19T22:16:52.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:51 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:49.906569+0000 mgr.smithi176.faoogw (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:52.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.243+0000 7fbc93aec700 1 -- 172.21.15.184:0/1615176618 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc8c108210 con 0x7fbc8c0f6510 2024-02-19T22:16:52.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.244+0000 7fbc827fc700 1 -- 172.21.15.184:0/1615176618 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbc7c01db70 con 0x7fbc8c0f6510 2024-02-19T22:16:52.246 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:52.246 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:52.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.245+0000 7fbc93aec700 1 -- 172.21.15.184:0/1615176618 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbc7802ff10 msgr2=0x7fbc780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:52.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.245+0000 7fbc93aec700 1 --2- 172.21.15.184:0/1615176618 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbc7802ff10 0x7fbc780323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fbc880096a0 tx=0x7fbc88006b40).stop 2024-02-19T22:16:52.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.245+0000 7fbc93aec700 1 -- 172.21.15.184:0/1615176618 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc8c0f6510 msgr2=0x7fbc8c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:52.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.245+0000 7fbc93aec700 1 --2- 172.21.15.184:0/1615176618 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c1067e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fbc7c005010 tx=0x7fbc7c00b040).stop 2024-02-19T22:16:52.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.245+0000 7fbc93aec700 1 -- 172.21.15.184:0/1615176618 shutdown_connections 2024-02-19T22:16:52.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.245+0000 7fbc93aec700 1 --2- 172.21.15.184:0/1615176618 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbc7802ff10 0x7fbc780323d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:52.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.245+0000 7fbc93aec700 1 --2- 172.21.15.184:0/1615176618 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c1067e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:52.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.245+0000 7fbc93aec700 1 -- 172.21.15.184:0/1615176618 >> 172.21.15.184:0/1615176618 conn(0x7fbc8c0f1c70 msgr2=0x7fbc8c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:52.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.245+0000 7fbc93aec700 1 -- 172.21.15.184:0/1615176618 shutdown_connections 2024-02-19T22:16:52.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:52.245+0000 7fbc93aec700 1 -- 172.21.15.184:0/1615176618 wait complete. 2024-02-19T22:16:52.248 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:53.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:52 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:52.245131+0000 mon.smithi176 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.184:0/1615176618' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:53.970 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:53.971 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:54.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:51.906874+0000 mgr.smithi176.faoogw (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:55.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.797+0000 7ff974e6d700 1 -- 172.21.15.184:0/1251403871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff970060160 msgr2=0x7ff9700f8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:55.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.797+0000 7ff974e6d700 1 --2- 172.21.15.184:0/1251403871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff970060160 0x7ff9700f8fa0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7ff960009fb0 tx=0x7ff960009b30).stop 2024-02-19T22:16:55.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.797+0000 7ff974e6d700 1 -- 172.21.15.184:0/1251403871 shutdown_connections 2024-02-19T22:16:55.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.797+0000 7ff974e6d700 1 --2- 172.21.15.184:0/1251403871 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff970060160 0x7ff9700f8fa0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:55.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.797+0000 7ff974e6d700 1 -- 172.21.15.184:0/1251403871 >> 172.21.15.184:0/1251403871 conn(0x7ff9700f09a0 msgr2=0x7ff9700f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:55.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.797+0000 7ff974e6d700 1 -- 172.21.15.184:0/1251403871 shutdown_connections 2024-02-19T22:16:55.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.797+0000 7ff974e6d700 1 -- 172.21.15.184:0/1251403871 wait complete. 2024-02-19T22:16:55.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.797+0000 7ff974e6d700 1 Processor -- start 2024-02-19T22:16:55.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.798+0000 7ff974e6d700 1 -- start start 2024-02-19T22:16:55.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.798+0000 7ff974e6d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff970060160 0x7ff9700f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:55.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.798+0000 7ff974e6d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9700f4ea0 con 0x7ff970060160 2024-02-19T22:16:55.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.798+0000 7ff96f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff970060160 0x7ff9700f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:55.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.798+0000 7ff96f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff970060160 0x7ff9700f6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:41520/0 (socket says 172.21.15.184:41520) 2024-02-19T22:16:55.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.798+0000 7ff96f7fe700 1 -- 172.21.15.184:0/3504384405 learned_addr learned my addr 172.21.15.184:0/3504384405 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:55.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.798+0000 7ff96f7fe700 1 -- 172.21.15.184:0/3504384405 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff960005040 con 0x7ff970060160 2024-02-19T22:16:55.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.798+0000 7ff96f7fe700 1 --2- 172.21.15.184:0/3504384405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff970060160 0x7ff9700f6a20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7ff960005010 tx=0x7ff96000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:55.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.799+0000 7ff96cff9700 1 -- 172.21.15.184:0/3504384405 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff960009320 con 0x7ff970060160 2024-02-19T22:16:55.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.799+0000 7ff974e6d700 1 -- 172.21.15.184:0/3504384405 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9700f5040 con 0x7ff970060160 2024-02-19T22:16:55.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.799+0000 7ff974e6d700 1 -- 172.21.15.184:0/3504384405 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9700f54d0 con 0x7ff970060160 2024-02-19T22:16:55.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.799+0000 7ff96cff9700 1 -- 172.21.15.184:0/3504384405 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff960013380 con 0x7ff970060160 2024-02-19T22:16:55.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.799+0000 7ff96cff9700 1 -- 172.21.15.184:0/3504384405 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff960014df0 con 0x7ff970060160 2024-02-19T22:16:55.803 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.799+0000 7ff974e6d700 1 -- 172.21.15.184:0/3504384405 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff970059100 con 0x7ff970060160 2024-02-19T22:16:55.803 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.800+0000 7ff96cff9700 1 -- 172.21.15.184:0/3504384405 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9600095f0 con 0x7ff970060160 2024-02-19T22:16:55.803 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.800+0000 7ff96cff9700 1 --2- 172.21.15.184:0/3504384405 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff95c02ff10 0x7ff95c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:55.803 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.800+0000 7ff96cff9700 1 -- 172.21.15.184:0/3504384405 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff960040eb0 con 0x7ff970060160 2024-02-19T22:16:55.803 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.800+0000 7ff966ffd700 1 --2- 172.21.15.184:0/3504384405 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff95c02ff10 0x7ff95c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:55.803 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.800+0000 7ff966ffd700 1 --2- 172.21.15.184:0/3504384405 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff95c02ff10 0x7ff95c0323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff9580096a0 tx=0x7ff958006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:55.806 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:55.804+0000 7ff96cff9700 1 -- 172.21.15.184:0/3504384405 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff96001dbb0 con 0x7ff970060160 2024-02-19T22:16:56.013 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:56.011+0000 7ff974e6d700 1 -- 172.21.15.184:0/3504384405 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9700fbe20 con 0x7ff970060160 2024-02-19T22:16:56.014 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:56.012+0000 7ff96cff9700 1 -- 172.21.15.184:0/3504384405 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff96001d9d0 con 0x7ff970060160 2024-02-19T22:16:56.014 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:56.014 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:56.015 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:56.014+0000 7ff974e6d700 1 -- 172.21.15.184:0/3504384405 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff95c02ff10 msgr2=0x7ff95c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:56.015 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:56.014+0000 7ff974e6d700 1 --2- 172.21.15.184:0/3504384405 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff95c02ff10 0x7ff95c0323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff9580096a0 tx=0x7ff958006b40).stop 2024-02-19T22:16:56.016 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:56.014+0000 7ff974e6d700 1 -- 172.21.15.184:0/3504384405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff970060160 msgr2=0x7ff9700f6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:56.016 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:56.014+0000 7ff974e6d700 1 --2- 172.21.15.184:0/3504384405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff970060160 0x7ff9700f6a20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7ff960005010 tx=0x7ff96000b040).stop 2024-02-19T22:16:56.016 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:56.014+0000 7ff974e6d700 1 -- 172.21.15.184:0/3504384405 shutdown_connections 2024-02-19T22:16:56.016 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:56.014+0000 7ff974e6d700 1 --2- 172.21.15.184:0/3504384405 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff95c02ff10 0x7ff95c0323d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:56.016 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:56.014+0000 7ff974e6d700 1 --2- 172.21.15.184:0/3504384405 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff970060160 0x7ff9700f6a20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:56.016 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:56.014+0000 7ff974e6d700 1 -- 172.21.15.184:0/3504384405 >> 172.21.15.184:0/3504384405 conn(0x7ff9700f09a0 msgr2=0x7ff9700f1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:56.016 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:56.014+0000 7ff974e6d700 1 -- 172.21.15.184:0/3504384405 shutdown_connections 2024-02-19T22:16:56.016 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:56.014+0000 7ff974e6d700 1 -- 172.21.15.184:0/3504384405 wait complete. 2024-02-19T22:16:56.016 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:16:56.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:55 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:53.907258+0000 mgr.smithi176.faoogw (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:57.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:56 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:56.013586+0000 mon.smithi176 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.184:0/3504384405' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:57.726 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:57.727 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:55.907646+0000 mgr.smithi176.faoogw (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:59.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.703+0000 7fe713a81700 1 -- 172.21.15.184:0/3302598273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe70c0f5410 msgr2=0x7fe70c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:59.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.703+0000 7fe713a81700 1 --2- 172.21.15.184:0/3302598273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe70c0f5410 0x7fe70c0f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fe6fc009fb0 tx=0x7fe6fc009b30).stop 2024-02-19T22:16:59.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.703+0000 7fe713a81700 1 -- 172.21.15.184:0/3302598273 shutdown_connections 2024-02-19T22:16:59.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.703+0000 7fe713a81700 1 --2- 172.21.15.184:0/3302598273 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe70c0f5410 0x7fe70c0f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:59.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.703+0000 7fe713a81700 1 -- 172.21.15.184:0/3302598273 >> 172.21.15.184:0/3302598273 conn(0x7fe70c0f0b70 msgr2=0x7fe70c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:59.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.704+0000 7fe713a81700 1 -- 172.21.15.184:0/3302598273 shutdown_connections 2024-02-19T22:16:59.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.704+0000 7fe713a81700 1 -- 172.21.15.184:0/3302598273 wait complete. 2024-02-19T22:16:59.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.704+0000 7fe713a81700 1 Processor -- start 2024-02-19T22:16:59.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.704+0000 7fe713a81700 1 -- start start 2024-02-19T22:16:59.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.704+0000 7fe713a81700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe70c0f5410 0x7fe70c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:59.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.704+0000 7fe713a81700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c105bd0 con 0x7fe70c0f5410 2024-02-19T22:16:59.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.705+0000 7fe71181d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe70c0f5410 0x7fe70c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:59.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.705+0000 7fe71181d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe70c0f5410 0x7fe70c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:48954/0 (socket says 172.21.15.184:48954) 2024-02-19T22:16:59.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.705+0000 7fe71181d700 1 -- 172.21.15.184:0/1754162303 learned_addr learned my addr 172.21.15.184:0/1754162303 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:16:59.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.705+0000 7fe71181d700 1 -- 172.21.15.184:0/1754162303 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc005040 con 0x7fe70c0f5410 2024-02-19T22:16:59.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.705+0000 7fe71181d700 1 --2- 172.21.15.184:0/1754162303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe70c0f5410 0x7fe70c1056e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fe6fc005010 tx=0x7fe6fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:59.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.705+0000 7fe7027fc700 1 -- 172.21.15.184:0/1754162303 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6fc009320 con 0x7fe70c0f5410 2024-02-19T22:16:59.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.705+0000 7fe713a81700 1 -- 172.21.15.184:0/1754162303 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe70c105dd0 con 0x7fe70c0f5410 2024-02-19T22:16:59.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.705+0000 7fe7027fc700 1 -- 172.21.15.184:0/1754162303 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6fc013380 con 0x7fe70c0f5410 2024-02-19T22:16:59.709 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.705+0000 7fe7027fc700 1 -- 172.21.15.184:0/1754162303 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6fc014e50 con 0x7fe70c0f5410 2024-02-19T22:16:59.709 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.705+0000 7fe713a81700 1 -- 172.21.15.184:0/1754162303 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70c106260 con 0x7fe70c0f5410 2024-02-19T22:16:59.709 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.706+0000 7fe7027fc700 1 -- 172.21.15.184:0/1754162303 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe6fc0095f0 con 0x7fe70c0f5410 2024-02-19T22:16:59.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.706+0000 7fe713a81700 1 -- 172.21.15.184:0/1754162303 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe70c045ad0 con 0x7fe70c0f5410 2024-02-19T22:16:59.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.706+0000 7fe7027fc700 1 --2- 172.21.15.184:0/1754162303 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe6f802ff10 0x7fe6f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:59.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.706+0000 7fe7027fc700 1 -- 172.21.15.184:0/1754162303 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe6fc040fa0 con 0x7fe70c0f5410 2024-02-19T22:16:59.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.706+0000 7fe71101c700 1 --2- 172.21.15.184:0/1754162303 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe6f802ff10 0x7fe6f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:59.710 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.707+0000 7fe71101c700 1 --2- 172.21.15.184:0/1754162303 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe6f802ff10 0x7fe6f80323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fe7080096a0 tx=0x7fe708006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:59.713 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.711+0000 7fe7027fc700 1 -- 172.21.15.184:0/1754162303 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe6fc014420 con 0x7fe70c0f5410 2024-02-19T22:16:59.922 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.920+0000 7fe713a81700 1 -- 172.21.15.184:0/1754162303 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe70c107110 con 0x7fe70c0f5410 2024-02-19T22:16:59.922 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.921+0000 7fe7027fc700 1 -- 172.21.15.184:0/1754162303 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe6fc010230 con 0x7fe70c0f5410 2024-02-19T22:16:59.923 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:16:59.923 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:59.924 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.922+0000 7fe713a81700 1 -- 172.21.15.184:0/1754162303 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe6f802ff10 msgr2=0x7fe6f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:59.924 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.922+0000 7fe713a81700 1 --2- 172.21.15.184:0/1754162303 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe6f802ff10 0x7fe6f80323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fe7080096a0 tx=0x7fe708006b40).stop 2024-02-19T22:16:59.924 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.922+0000 7fe713a81700 1 -- 172.21.15.184:0/1754162303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe70c0f5410 msgr2=0x7fe70c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:59.924 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.922+0000 7fe713a81700 1 --2- 172.21.15.184:0/1754162303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe70c0f5410 0x7fe70c1056e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fe6fc005010 tx=0x7fe6fc00b040).stop 2024-02-19T22:16:59.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.922+0000 7fe713a81700 1 -- 172.21.15.184:0/1754162303 shutdown_connections 2024-02-19T22:16:59.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.922+0000 7fe713a81700 1 --2- 172.21.15.184:0/1754162303 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe6f802ff10 0x7fe6f80323d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:59.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.922+0000 7fe713a81700 1 --2- 172.21.15.184:0/1754162303 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe70c0f5410 0x7fe70c1056e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:59.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.922+0000 7fe713a81700 1 -- 172.21.15.184:0/1754162303 >> 172.21.15.184:0/1754162303 conn(0x7fe70c0f0b70 msgr2=0x7fe70c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:59.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.922+0000 7fe713a81700 1 -- 172.21.15.184:0/1754162303 shutdown_connections 2024-02-19T22:16:59.925 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:16:59.922+0000 7fe713a81700 1 -- 172.21.15.184:0/1754162303 wait complete. 2024-02-19T22:16:59.925 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:00.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:16:59 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:57.907986+0000 mgr.smithi176.faoogw (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:01.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:00 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:16:59.922076+0000 mon.smithi176 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.184:0/1754162303' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:01.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:01.647 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:02.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:16:59.908350+0000 mgr.smithi176.faoogw (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:03.571 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.567+0000 7fd4e800a700 1 -- 172.21.15.184:0/3693271308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4e00f4740 msgr2=0x7fd4e00f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:03.571 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.567+0000 7fd4e800a700 1 --2- 172.21.15.184:0/3693271308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4e00f4740 0x7fd4e00f4b60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fd4d0009fb0 tx=0x7fd4d0009b30).stop 2024-02-19T22:17:03.571 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.567+0000 7fd4e800a700 1 -- 172.21.15.184:0/3693271308 shutdown_connections 2024-02-19T22:17:03.571 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.567+0000 7fd4e800a700 1 --2- 172.21.15.184:0/3693271308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4e00f4740 0x7fd4e00f4b60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:03.571 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.567+0000 7fd4e800a700 1 -- 172.21.15.184:0/3693271308 >> 172.21.15.184:0/3693271308 conn(0x7fd4e00f0260 msgr2=0x7fd4e00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:03.571 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.567+0000 7fd4e800a700 1 -- 172.21.15.184:0/3693271308 shutdown_connections 2024-02-19T22:17:03.571 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.567+0000 7fd4e800a700 1 -- 172.21.15.184:0/3693271308 wait complete. 2024-02-19T22:17:03.571 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.568+0000 7fd4e800a700 1 Processor -- start 2024-02-19T22:17:03.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.568+0000 7fd4e800a700 1 -- start start 2024-02-19T22:17:03.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.568+0000 7fd4e800a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4e00f4740 0x7fd4e01067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:03.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.568+0000 7fd4e800a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e0106ca0 con 0x7fd4e00f4740 2024-02-19T22:17:03.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.568+0000 7fd4e5da6700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4e00f4740 0x7fd4e01067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:03.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.568+0000 7fd4e5da6700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4e00f4740 0x7fd4e01067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:48968/0 (socket says 172.21.15.184:48968) 2024-02-19T22:17:03.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.568+0000 7fd4e5da6700 1 -- 172.21.15.184:0/2793730123 learned_addr learned my addr 172.21.15.184:0/2793730123 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:03.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.568+0000 7fd4e5da6700 1 -- 172.21.15.184:0/2793730123 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d0005040 con 0x7fd4e00f4740 2024-02-19T22:17:03.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.569+0000 7fd4e5da6700 1 --2- 172.21.15.184:0/2793730123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4e00f4740 0x7fd4e01067b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fd4d0005010 tx=0x7fd4d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:03.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.569+0000 7fd4d67fc700 1 -- 172.21.15.184:0/2793730123 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4d0009320 con 0x7fd4e00f4740 2024-02-19T22:17:03.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.569+0000 7fd4d67fc700 1 -- 172.21.15.184:0/2793730123 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4d0013380 con 0x7fd4e00f4740 2024-02-19T22:17:03.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.569+0000 7fd4e800a700 1 -- 172.21.15.184:0/2793730123 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e0106ea0 con 0x7fd4e00f4740 2024-02-19T22:17:03.572 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.569+0000 7fd4d67fc700 1 -- 172.21.15.184:0/2793730123 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4d0014e50 con 0x7fd4e00f4740 2024-02-19T22:17:03.573 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.569+0000 7fd4e800a700 1 -- 172.21.15.184:0/2793730123 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e0107330 con 0x7fd4e00f4740 2024-02-19T22:17:03.573 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.570+0000 7fd4d67fc700 1 -- 172.21.15.184:0/2793730123 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4d00095f0 con 0x7fd4e00f4740 2024-02-19T22:17:03.573 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.570+0000 7fd4e800a700 1 -- 172.21.15.184:0/2793730123 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e0045ad0 con 0x7fd4e00f4740 2024-02-19T22:17:03.574 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.570+0000 7fd4d67fc700 1 --2- 172.21.15.184:0/2793730123 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd4cc02ff10 0x7fd4cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:03.574 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.570+0000 7fd4d67fc700 1 -- 172.21.15.184:0/2793730123 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4d0040f60 con 0x7fd4e00f4740 2024-02-19T22:17:03.574 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.570+0000 7fd4e55a5700 1 --2- 172.21.15.184:0/2793730123 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd4cc02ff10 0x7fd4cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:03.574 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.570+0000 7fd4e55a5700 1 --2- 172.21.15.184:0/2793730123 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd4cc02ff10 0x7fd4cc0323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fd4dc0096a0 tx=0x7fd4dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:03.576 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.574+0000 7fd4d67fc700 1 -- 172.21.15.184:0/2793730123 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4d0014420 con 0x7fd4e00f4740 2024-02-19T22:17:03.780 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.778+0000 7fd4e800a700 1 -- 172.21.15.184:0/2793730123 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4e01081e0 con 0x7fd4e00f4740 2024-02-19T22:17:03.781 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.779+0000 7fd4d67fc700 1 -- 172.21.15.184:0/2793730123 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd4d0014420 con 0x7fd4e00f4740 2024-02-19T22:17:03.781 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:03.781 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:03.782 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.780+0000 7fd4e800a700 1 -- 172.21.15.184:0/2793730123 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd4cc02ff10 msgr2=0x7fd4cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:03.782 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.780+0000 7fd4e800a700 1 --2- 172.21.15.184:0/2793730123 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd4cc02ff10 0x7fd4cc0323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fd4dc0096a0 tx=0x7fd4dc006b40).stop 2024-02-19T22:17:03.783 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.780+0000 7fd4e800a700 1 -- 172.21.15.184:0/2793730123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4e00f4740 msgr2=0x7fd4e01067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:03.783 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.780+0000 7fd4e800a700 1 --2- 172.21.15.184:0/2793730123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4e00f4740 0x7fd4e01067b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fd4d0005010 tx=0x7fd4d000b040).stop 2024-02-19T22:17:03.783 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.780+0000 7fd4e800a700 1 -- 172.21.15.184:0/2793730123 shutdown_connections 2024-02-19T22:17:03.783 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.780+0000 7fd4e800a700 1 --2- 172.21.15.184:0/2793730123 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd4cc02ff10 0x7fd4cc0323d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:03.783 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.780+0000 7fd4e800a700 1 --2- 172.21.15.184:0/2793730123 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd4e00f4740 0x7fd4e01067b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:03.783 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.780+0000 7fd4e800a700 1 -- 172.21.15.184:0/2793730123 >> 172.21.15.184:0/2793730123 conn(0x7fd4e00f0260 msgr2=0x7fd4e00f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:03.783 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.780+0000 7fd4e800a700 1 -- 172.21.15.184:0/2793730123 shutdown_connections 2024-02-19T22:17:03.783 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:03.780+0000 7fd4e800a700 1 -- 172.21.15.184:0/2793730123 wait complete. 2024-02-19T22:17:03.783 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:04.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:01.908702+0000 mgr.smithi176.faoogw (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:04.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:03.780179+0000 mon.smithi176 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.184:0/2793730123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:05.444 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:05.445 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:06.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:03.908995+0000 mgr.smithi176.faoogw (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:07.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.501+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1339935449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f980601e0 msgr2=0x7f3f980f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:07.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.501+0000 7f3f9ebff700 1 --2- 172.21.15.184:0/1339935449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f980601e0 0x7f3f980f8f20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f3f88009fb0 tx=0x7f3f88009b30).stop 2024-02-19T22:17:07.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.501+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1339935449 shutdown_connections 2024-02-19T22:17:07.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.501+0000 7f3f9ebff700 1 --2- 172.21.15.184:0/1339935449 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f980601e0 0x7f3f980f8f20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:07.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.501+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1339935449 >> 172.21.15.184:0/1339935449 conn(0x7f3f980f09a0 msgr2=0x7f3f980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:07.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.501+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1339935449 shutdown_connections 2024-02-19T22:17:07.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.501+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1339935449 wait complete. 2024-02-19T22:17:07.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.501+0000 7f3f9ebff700 1 Processor -- start 2024-02-19T22:17:07.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.502+0000 7f3f9ebff700 1 -- start start 2024-02-19T22:17:07.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.502+0000 7f3f9ebff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f980601e0 0x7f3f980f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:07.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.502+0000 7f3f9ebff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f980f4e20 con 0x7f3f980601e0 2024-02-19T22:17:07.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.502+0000 7f3f9dbfd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f980601e0 0x7f3f980f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:07.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.502+0000 7f3f9dbfd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f980601e0 0x7f3f980f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:33122/0 (socket says 172.21.15.184:33122) 2024-02-19T22:17:07.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.502+0000 7f3f9dbfd700 1 -- 172.21.15.184:0/1881330652 learned_addr learned my addr 172.21.15.184:0/1881330652 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:07.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.502+0000 7f3f9dbfd700 1 -- 172.21.15.184:0/1881330652 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f88005040 con 0x7f3f980601e0 2024-02-19T22:17:07.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.502+0000 7f3f9dbfd700 1 --2- 172.21.15.184:0/1881330652 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f980601e0 0x7f3f980f69a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f3f88005010 tx=0x7f3f8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:07.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.502+0000 7f3f8e7fc700 1 -- 172.21.15.184:0/1881330652 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f88009320 con 0x7f3f980601e0 2024-02-19T22:17:07.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.502+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1881330652 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f980f4fc0 con 0x7f3f980601e0 2024-02-19T22:17:07.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.502+0000 7f3f8e7fc700 1 -- 172.21.15.184:0/1881330652 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f88013380 con 0x7f3f980601e0 2024-02-19T22:17:07.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.502+0000 7f3f8e7fc700 1 -- 172.21.15.184:0/1881330652 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f88014e50 con 0x7f3f980601e0 2024-02-19T22:17:07.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.502+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1881330652 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f980f5450 con 0x7f3f980601e0 2024-02-19T22:17:07.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.503+0000 7f3f8e7fc700 1 -- 172.21.15.184:0/1881330652 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f880095f0 con 0x7f3f980601e0 2024-02-19T22:17:07.517 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.503+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1881330652 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f98059100 con 0x7f3f980601e0 2024-02-19T22:17:07.518 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.503+0000 7f3f8e7fc700 1 --2- 172.21.15.184:0/1881330652 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3f8402ff10 0x7f3f840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:07.518 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.503+0000 7f3f8e7fc700 1 -- 172.21.15.184:0/1881330652 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f88040fa0 con 0x7f3f980601e0 2024-02-19T22:17:07.518 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.504+0000 7f3f9d3fc700 1 --2- 172.21.15.184:0/1881330652 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3f8402ff10 0x7f3f840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:07.518 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.504+0000 7f3f9d3fc700 1 --2- 172.21.15.184:0/1881330652 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3f8402ff10 0x7f3f840323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f3f940096a0 tx=0x7f3f94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:07.518 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.508+0000 7f3f8e7fc700 1 -- 172.21.15.184:0/1881330652 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f88014420 con 0x7f3f980601e0 2024-02-19T22:17:07.718 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.717+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1881330652 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f980fbe10 con 0x7f3f980601e0 2024-02-19T22:17:07.719 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.718+0000 7f3f8e7fc700 1 -- 172.21.15.184:0/1881330652 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3f88010230 con 0x7f3f980601e0 2024-02-19T22:17:07.720 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:07.720 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:07.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.719+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1881330652 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3f8402ff10 msgr2=0x7f3f840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:07.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.719+0000 7f3f9ebff700 1 --2- 172.21.15.184:0/1881330652 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3f8402ff10 0x7f3f840323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f3f940096a0 tx=0x7f3f94006b40).stop 2024-02-19T22:17:07.721 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.719+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1881330652 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f980601e0 msgr2=0x7f3f980f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:07.722 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.719+0000 7f3f9ebff700 1 --2- 172.21.15.184:0/1881330652 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f980601e0 0x7f3f980f69a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f3f88005010 tx=0x7f3f8800b040).stop 2024-02-19T22:17:07.722 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.719+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1881330652 shutdown_connections 2024-02-19T22:17:07.722 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.719+0000 7f3f9ebff700 1 --2- 172.21.15.184:0/1881330652 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3f8402ff10 0x7f3f840323d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:07.722 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.719+0000 7f3f9ebff700 1 --2- 172.21.15.184:0/1881330652 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3f980601e0 0x7f3f980f69a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:07.722 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.719+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1881330652 >> 172.21.15.184:0/1881330652 conn(0x7f3f980f09a0 msgr2=0x7f3f980f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:07.722 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.719+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1881330652 shutdown_connections 2024-02-19T22:17:07.722 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:07.719+0000 7f3f9ebff700 1 -- 172.21.15.184:0/1881330652 wait complete. 2024-02-19T22:17:07.722 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:08.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:05.909347+0000 mgr.smithi176.faoogw (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:08.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:07.718831+0000 mon.smithi176 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.184:0/1881330652' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:09.417 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:09.417 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:10.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:07.909716+0000 mgr.smithi176.faoogw (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:11.280 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.277+0000 7f932f4d3700 1 -- 172.21.15.184:0/3602624214 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93280f5410 msgr2=0x7f93280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:11.280 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.277+0000 7f932f4d3700 1 --2- 172.21.15.184:0/3602624214 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93280f5410 0x7f93280f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f9324009fb0 tx=0x7f9324009b30).stop 2024-02-19T22:17:11.280 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.277+0000 7f932f4d3700 1 -- 172.21.15.184:0/3602624214 shutdown_connections 2024-02-19T22:17:11.280 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.277+0000 7f932f4d3700 1 --2- 172.21.15.184:0/3602624214 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93280f5410 0x7f93280f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:11.280 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.277+0000 7f932f4d3700 1 -- 172.21.15.184:0/3602624214 >> 172.21.15.184:0/3602624214 conn(0x7f93280f0b70 msgr2=0x7f93280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:11.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.277+0000 7f932f4d3700 1 -- 172.21.15.184:0/3602624214 shutdown_connections 2024-02-19T22:17:11.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.277+0000 7f932f4d3700 1 -- 172.21.15.184:0/3602624214 wait complete. 2024-02-19T22:17:11.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.277+0000 7f932f4d3700 1 Processor -- start 2024-02-19T22:17:11.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.277+0000 7f932f4d3700 1 -- start start 2024-02-19T22:17:11.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.277+0000 7f932f4d3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93280f5410 0x7f93281094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:11.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.277+0000 7f932f4d3700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9328060240 con 0x7f93280f5410 2024-02-19T22:17:11.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.278+0000 7f932d26f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93280f5410 0x7f93281094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:11.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.278+0000 7f932d26f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93280f5410 0x7f93281094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:33138/0 (socket says 172.21.15.184:33138) 2024-02-19T22:17:11.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.278+0000 7f932d26f700 1 -- 172.21.15.184:0/1503844941 learned_addr learned my addr 172.21.15.184:0/1503844941 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:11.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.278+0000 7f932d26f700 1 -- 172.21.15.184:0/1503844941 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9324005040 con 0x7f93280f5410 2024-02-19T22:17:11.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.278+0000 7f932d26f700 1 --2- 172.21.15.184:0/1503844941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93280f5410 0x7f93281094e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f93240089f0 tx=0x7f932400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:11.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.278+0000 7f9319ffb700 1 -- 172.21.15.184:0/1503844941 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9324009320 con 0x7f93280f5410 2024-02-19T22:17:11.281 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.278+0000 7f932f4d3700 1 -- 172.21.15.184:0/1503844941 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9328060440 con 0x7f93280f5410 2024-02-19T22:17:11.282 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.278+0000 7f9319ffb700 1 -- 172.21.15.184:0/1503844941 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9324013380 con 0x7f93280f5410 2024-02-19T22:17:11.282 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.278+0000 7f9319ffb700 1 -- 172.21.15.184:0/1503844941 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f932401ed80 con 0x7f93280f5410 2024-02-19T22:17:11.282 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.278+0000 7f932f4d3700 1 -- 172.21.15.184:0/1503844941 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9328109b90 con 0x7f93280f5410 2024-02-19T22:17:11.282 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.279+0000 7f9319ffb700 1 -- 172.21.15.184:0/1503844941 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f932401e420 con 0x7f93280f5410 2024-02-19T22:17:11.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.279+0000 7f932f4d3700 1 -- 172.21.15.184:0/1503844941 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9328045ad0 con 0x7f93280f5410 2024-02-19T22:17:11.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.279+0000 7f9319ffb700 1 --2- 172.21.15.184:0/1503844941 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f931402ff60 0x7f9314032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:11.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.279+0000 7f9319ffb700 1 -- 172.21.15.184:0/1503844941 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9324041120 con 0x7f93280f5410 2024-02-19T22:17:11.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.279+0000 7f932ca6e700 1 --2- 172.21.15.184:0/1503844941 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f931402ff60 0x7f9314032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:11.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.280+0000 7f932ca6e700 1 --2- 172.21.15.184:0/1503844941 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f931402ff60 0x7f9314032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f931c0096a0 tx=0x7f931c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:11.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.284+0000 7f9319ffb700 1 -- 172.21.15.184:0/1503844941 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9324014cc0 con 0x7f93280f5410 2024-02-19T22:17:11.489 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.488+0000 7f932f4d3700 1 -- 172.21.15.184:0/1503844941 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f932810a3a0 con 0x7f93280f5410 2024-02-19T22:17:11.490 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.488+0000 7f9319ffb700 1 -- 172.21.15.184:0/1503844941 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f932402c2d0 con 0x7f93280f5410 2024-02-19T22:17:11.490 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:11.490 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:11.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.490+0000 7f932f4d3700 1 -- 172.21.15.184:0/1503844941 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f931402ff60 msgr2=0x7f9314032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:11.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.490+0000 7f932f4d3700 1 --2- 172.21.15.184:0/1503844941 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f931402ff60 0x7f9314032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f931c0096a0 tx=0x7f931c006b40).stop 2024-02-19T22:17:11.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.490+0000 7f932f4d3700 1 -- 172.21.15.184:0/1503844941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93280f5410 msgr2=0x7f93281094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:11.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.490+0000 7f932f4d3700 1 --2- 172.21.15.184:0/1503844941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93280f5410 0x7f93281094e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f93240089f0 tx=0x7f932400b040).stop 2024-02-19T22:17:11.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.490+0000 7f932f4d3700 1 -- 172.21.15.184:0/1503844941 shutdown_connections 2024-02-19T22:17:11.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.490+0000 7f932f4d3700 1 --2- 172.21.15.184:0/1503844941 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f931402ff60 0x7f9314032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:11.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.490+0000 7f932f4d3700 1 --2- 172.21.15.184:0/1503844941 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f93280f5410 0x7f93281094e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:11.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.490+0000 7f932f4d3700 1 -- 172.21.15.184:0/1503844941 >> 172.21.15.184:0/1503844941 conn(0x7f93280f0b70 msgr2=0x7f93280f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:11.492 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.490+0000 7f932f4d3700 1 -- 172.21.15.184:0/1503844941 shutdown_connections 2024-02-19T22:17:11.493 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:11.490+0000 7f932f4d3700 1 -- 172.21.15.184:0/1503844941 wait complete. 2024-02-19T22:17:11.493 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:12.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:09.910053+0000 mgr.smithi176.faoogw (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:12.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:11.489655+0000 mon.smithi176 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.184:0/1503844941' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:13.229 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:13.230 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:14.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:13 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:11.910342+0000 mgr.smithi176.faoogw (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:15.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.164+0000 7fbd8f59e700 1 -- 172.21.15.184:0/3256542334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd900601e0 msgr2=0x7fbd900f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:15.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.164+0000 7fbd8f59e700 1 --2- 172.21.15.184:0/3256542334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd900601e0 0x7fbd900f8f20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fbd80009fb0 tx=0x7fbd80009b30).stop 2024-02-19T22:17:15.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.165+0000 7fbd8f59e700 1 -- 172.21.15.184:0/3256542334 shutdown_connections 2024-02-19T22:17:15.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.165+0000 7fbd8f59e700 1 --2- 172.21.15.184:0/3256542334 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd900601e0 0x7fbd900f8f20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:15.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.165+0000 7fbd8f59e700 1 -- 172.21.15.184:0/3256542334 >> 172.21.15.184:0/3256542334 conn(0x7fbd900f09a0 msgr2=0x7fbd900f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:15.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.165+0000 7fbd8f59e700 1 -- 172.21.15.184:0/3256542334 shutdown_connections 2024-02-19T22:17:15.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.165+0000 7fbd8f59e700 1 -- 172.21.15.184:0/3256542334 wait complete. 2024-02-19T22:17:15.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.165+0000 7fbd8f59e700 1 Processor -- start 2024-02-19T22:17:15.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.165+0000 7fbd8f59e700 1 -- start start 2024-02-19T22:17:15.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.165+0000 7fbd8f59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd900601e0 0x7fbd900f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:15.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.165+0000 7fbd8f59e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd900f4e20 con 0x7fbd900601e0 2024-02-19T22:17:15.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.165+0000 7fbd8e59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd900601e0 0x7fbd900f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:15.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.165+0000 7fbd8e59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd900601e0 0x7fbd900f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:33156/0 (socket says 172.21.15.184:33156) 2024-02-19T22:17:15.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.165+0000 7fbd8e59c700 1 -- 172.21.15.184:0/2554980862 learned_addr learned my addr 172.21.15.184:0/2554980862 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:15.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.166+0000 7fbd8e59c700 1 -- 172.21.15.184:0/2554980862 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd80005040 con 0x7fbd900601e0 2024-02-19T22:17:15.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.166+0000 7fbd8e59c700 1 --2- 172.21.15.184:0/2554980862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd900601e0 0x7fbd900f69a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fbd80005010 tx=0x7fbd8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:15.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.166+0000 7fbd7effd700 1 -- 172.21.15.184:0/2554980862 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd80009320 con 0x7fbd900601e0 2024-02-19T22:17:15.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.166+0000 7fbd8f59e700 1 -- 172.21.15.184:0/2554980862 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd900f4fc0 con 0x7fbd900601e0 2024-02-19T22:17:15.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.166+0000 7fbd7effd700 1 -- 172.21.15.184:0/2554980862 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd80013380 con 0x7fbd900601e0 2024-02-19T22:17:15.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.166+0000 7fbd7effd700 1 -- 172.21.15.184:0/2554980862 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd80014e50 con 0x7fbd900601e0 2024-02-19T22:17:15.170 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.166+0000 7fbd8f59e700 1 -- 172.21.15.184:0/2554980862 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd900f5450 con 0x7fbd900601e0 2024-02-19T22:17:15.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.167+0000 7fbd7effd700 1 -- 172.21.15.184:0/2554980862 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd800095f0 con 0x7fbd900601e0 2024-02-19T22:17:15.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.167+0000 7fbd8f59e700 1 -- 172.21.15.184:0/2554980862 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd90059100 con 0x7fbd900601e0 2024-02-19T22:17:15.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.167+0000 7fbd7effd700 1 --2- 172.21.15.184:0/2554980862 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbd7802ff10 0x7fbd780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:15.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.167+0000 7fbd7effd700 1 -- 172.21.15.184:0/2554980862 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd80040e70 con 0x7fbd900601e0 2024-02-19T22:17:15.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.167+0000 7fbd8dd9b700 1 --2- 172.21.15.184:0/2554980862 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbd7802ff10 0x7fbd780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:15.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.168+0000 7fbd8dd9b700 1 --2- 172.21.15.184:0/2554980862 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbd7802ff10 0x7fbd780323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fbd840096a0 tx=0x7fbd84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:15.173 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.171+0000 7fbd7effd700 1 -- 172.21.15.184:0/2554980862 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd80014420 con 0x7fbd900601e0 2024-02-19T22:17:15.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.374+0000 7fbd8f59e700 1 -- 172.21.15.184:0/2554980862 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd90045ad0 con 0x7fbd900601e0 2024-02-19T22:17:15.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.374+0000 7fbd7effd700 1 -- 172.21.15.184:0/2554980862 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbd80010230 con 0x7fbd900601e0 2024-02-19T22:17:15.377 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:15.377 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:15.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.376+0000 7fbd8f59e700 1 -- 172.21.15.184:0/2554980862 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbd7802ff10 msgr2=0x7fbd780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:15.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.376+0000 7fbd8f59e700 1 --2- 172.21.15.184:0/2554980862 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbd7802ff10 0x7fbd780323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fbd840096a0 tx=0x7fbd84006b40).stop 2024-02-19T22:17:15.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.376+0000 7fbd8f59e700 1 -- 172.21.15.184:0/2554980862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd900601e0 msgr2=0x7fbd900f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:15.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.376+0000 7fbd8f59e700 1 --2- 172.21.15.184:0/2554980862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd900601e0 0x7fbd900f69a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fbd80005010 tx=0x7fbd8000b040).stop 2024-02-19T22:17:15.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.376+0000 7fbd8f59e700 1 -- 172.21.15.184:0/2554980862 shutdown_connections 2024-02-19T22:17:15.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.376+0000 7fbd8f59e700 1 --2- 172.21.15.184:0/2554980862 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbd7802ff10 0x7fbd780323d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:15.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.376+0000 7fbd8f59e700 1 --2- 172.21.15.184:0/2554980862 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd900601e0 0x7fbd900f69a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:15.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.376+0000 7fbd8f59e700 1 -- 172.21.15.184:0/2554980862 >> 172.21.15.184:0/2554980862 conn(0x7fbd900f09a0 msgr2=0x7fbd900f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:15.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.376+0000 7fbd8f59e700 1 -- 172.21.15.184:0/2554980862 shutdown_connections 2024-02-19T22:17:15.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:15.376+0000 7fbd8f59e700 1 -- 172.21.15.184:0/2554980862 wait complete. 2024-02-19T22:17:15.379 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:16.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:13.910677+0000 mgr.smithi176.faoogw (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:16.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:15.375727+0000 mon.smithi176 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.184:0/2554980862' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:17.048 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:17.048 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:18.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:17 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:15.910959+0000 mgr.smithi176.faoogw (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:18.892 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.889+0000 7f88db7a5700 1 -- 172.21.15.184:0/2174289177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88d40f6510 msgr2=0x7f88d40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:18.892 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.889+0000 7f88db7a5700 1 --2- 172.21.15.184:0/2174289177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88d40f6510 0x7f88d40f6930 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f88c4009fb0 tx=0x7f88c4009b30).stop 2024-02-19T22:17:18.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.889+0000 7f88db7a5700 1 -- 172.21.15.184:0/2174289177 shutdown_connections 2024-02-19T22:17:18.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.889+0000 7f88db7a5700 1 --2- 172.21.15.184:0/2174289177 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88d40f6510 0x7f88d40f6930 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:18.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.889+0000 7f88db7a5700 1 -- 172.21.15.184:0/2174289177 >> 172.21.15.184:0/2174289177 conn(0x7f88d40f1c70 msgr2=0x7f88d40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:18.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.889+0000 7f88db7a5700 1 -- 172.21.15.184:0/2174289177 shutdown_connections 2024-02-19T22:17:18.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.889+0000 7f88db7a5700 1 -- 172.21.15.184:0/2174289177 wait complete. 2024-02-19T22:17:18.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.889+0000 7f88db7a5700 1 Processor -- start 2024-02-19T22:17:18.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.890+0000 7f88db7a5700 1 -- start start 2024-02-19T22:17:18.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.890+0000 7f88db7a5700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88d40f6510 0x7f88d406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:18.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.890+0000 7f88db7a5700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d4069e20 con 0x7f88d40f6510 2024-02-19T22:17:18.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.890+0000 7f88d9541700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88d40f6510 0x7f88d406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:18.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.890+0000 7f88d9541700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88d40f6510 0x7f88d406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:57704/0 (socket says 172.21.15.184:57704) 2024-02-19T22:17:18.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.890+0000 7f88d9541700 1 -- 172.21.15.184:0/3481174050 learned_addr learned my addr 172.21.15.184:0/3481174050 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:18.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.890+0000 7f88d9541700 1 -- 172.21.15.184:0/3481174050 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c4005040 con 0x7f88d40f6510 2024-02-19T22:17:18.893 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.890+0000 7f88d9541700 1 --2- 172.21.15.184:0/3481174050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88d40f6510 0x7f88d406b9a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f88c4005010 tx=0x7f88c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:18.894 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.890+0000 7f88c9ffb700 1 -- 172.21.15.184:0/3481174050 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88c4009320 con 0x7f88d40f6510 2024-02-19T22:17:18.894 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.890+0000 7f88db7a5700 1 -- 172.21.15.184:0/3481174050 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d4069fc0 con 0x7f88d40f6510 2024-02-19T22:17:18.894 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.890+0000 7f88c9ffb700 1 -- 172.21.15.184:0/3481174050 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f88c4013380 con 0x7f88d40f6510 2024-02-19T22:17:18.894 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.890+0000 7f88c9ffb700 1 -- 172.21.15.184:0/3481174050 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88c4014e50 con 0x7f88d40f6510 2024-02-19T22:17:18.894 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.891+0000 7f88db7a5700 1 -- 172.21.15.184:0/3481174050 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d406a450 con 0x7f88d40f6510 2024-02-19T22:17:18.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.891+0000 7f88c9ffb700 1 -- 172.21.15.184:0/3481174050 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88c40263a0 con 0x7f88d40f6510 2024-02-19T22:17:18.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.892+0000 7f88c9ffb700 1 --2- 172.21.15.184:0/3481174050 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f88c002ff10 0x7f88c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:18.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.892+0000 7f88c9ffb700 1 -- 172.21.15.184:0/3481174050 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88c4040f20 con 0x7f88d40f6510 2024-02-19T22:17:18.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.892+0000 7f88db7a5700 1 -- 172.21.15.184:0/3481174050 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d4059100 con 0x7f88d40f6510 2024-02-19T22:17:18.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.892+0000 7f88d8d40700 1 --2- 172.21.15.184:0/3481174050 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f88c002ff10 0x7f88c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:18.895 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.892+0000 7f88d8d40700 1 --2- 172.21.15.184:0/3481174050 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f88c002ff10 0x7f88c00323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f88d00096a0 tx=0x7f88d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:18.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:18.897+0000 7f88c9ffb700 1 -- 172.21.15.184:0/3481174050 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88c4012920 con 0x7f88d40f6510 2024-02-19T22:17:19.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:19.102+0000 7f88db7a5700 1 -- 172.21.15.184:0/3481174050 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88d40fc0a0 con 0x7f88d40f6510 2024-02-19T22:17:19.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:19.102+0000 7f88c9ffb700 1 -- 172.21.15.184:0/3481174050 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f88c4012330 con 0x7f88d40f6510 2024-02-19T22:17:19.104 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:19.104 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:19.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:19.104+0000 7f88db7a5700 1 -- 172.21.15.184:0/3481174050 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f88c002ff10 msgr2=0x7f88c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:19.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:19.104+0000 7f88db7a5700 1 --2- 172.21.15.184:0/3481174050 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f88c002ff10 0x7f88c00323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f88d00096a0 tx=0x7f88d0006b40).stop 2024-02-19T22:17:19.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:19.104+0000 7f88db7a5700 1 -- 172.21.15.184:0/3481174050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88d40f6510 msgr2=0x7f88d406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:19.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:19.104+0000 7f88db7a5700 1 --2- 172.21.15.184:0/3481174050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88d40f6510 0x7f88d406b9a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f88c4005010 tx=0x7f88c400b040).stop 2024-02-19T22:17:19.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:19.104+0000 7f88db7a5700 1 -- 172.21.15.184:0/3481174050 shutdown_connections 2024-02-19T22:17:19.107 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:19.104+0000 7f88db7a5700 1 --2- 172.21.15.184:0/3481174050 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f88c002ff10 0x7f88c00323d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:19.107 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:19.104+0000 7f88db7a5700 1 --2- 172.21.15.184:0/3481174050 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f88d40f6510 0x7f88d406b9a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:19.107 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:19.104+0000 7f88db7a5700 1 -- 172.21.15.184:0/3481174050 >> 172.21.15.184:0/3481174050 conn(0x7f88d40f1c70 msgr2=0x7f88d40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:19.107 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:19.104+0000 7f88db7a5700 1 -- 172.21.15.184:0/3481174050 shutdown_connections 2024-02-19T22:17:19.107 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:19.104+0000 7f88db7a5700 1 -- 172.21.15.184:0/3481174050 wait complete. 2024-02-19T22:17:19.107 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:20.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:17.911313+0000 mgr.smithi176.faoogw (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:20.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:19.103664+0000 mon.smithi176 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.184:0/3481174050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:20.855 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:20.855 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:22.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:19.911630+0000 mgr.smithi176.faoogw (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:22.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.657+0000 7f7594a7d700 1 -- 172.21.15.184:0/1697879119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900f51a0 msgr2=0x7f75900f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:22.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.657+0000 7f7594a7d700 1 --2- 172.21.15.184:0/1697879119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900f51a0 0x7f75900f55c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f7580009fb0 tx=0x7f7580009b30).stop 2024-02-19T22:17:22.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.658+0000 7f7594a7d700 1 -- 172.21.15.184:0/1697879119 shutdown_connections 2024-02-19T22:17:22.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.658+0000 7f7594a7d700 1 --2- 172.21.15.184:0/1697879119 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900f51a0 0x7f75900f55c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:22.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.658+0000 7f7594a7d700 1 -- 172.21.15.184:0/1697879119 >> 172.21.15.184:0/1697879119 conn(0x7f75900f09a0 msgr2=0x7f75900f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:22.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.658+0000 7f7594a7d700 1 -- 172.21.15.184:0/1697879119 shutdown_connections 2024-02-19T22:17:22.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.658+0000 7f7594a7d700 1 -- 172.21.15.184:0/1697879119 wait complete. 2024-02-19T22:17:22.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.658+0000 7f7594a7d700 1 Processor -- start 2024-02-19T22:17:22.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.658+0000 7f7594a7d700 1 -- start start 2024-02-19T22:17:22.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.658+0000 7f7594a7d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900f51a0 0x7f75901092c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:22.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.658+0000 7f7594a7d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7590060220 con 0x7f75900f51a0 2024-02-19T22:17:22.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.659+0000 7f758f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900f51a0 0x7f75901092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:22.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.659+0000 7f758f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900f51a0 0x7f75901092c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:57720/0 (socket says 172.21.15.184:57720) 2024-02-19T22:17:22.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.659+0000 7f758f7fe700 1 -- 172.21.15.184:0/3062600162 learned_addr learned my addr 172.21.15.184:0/3062600162 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:22.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.659+0000 7f758f7fe700 1 -- 172.21.15.184:0/3062600162 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7580005040 con 0x7f75900f51a0 2024-02-19T22:17:22.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.659+0000 7f758f7fe700 1 --2- 172.21.15.184:0/3062600162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900f51a0 0x7f75901092c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f75800089f0 tx=0x7f758000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:22.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.659+0000 7f7577fff700 1 -- 172.21.15.184:0/3062600162 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7580009320 con 0x7f75900f51a0 2024-02-19T22:17:22.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.660+0000 7f7594a7d700 1 -- 172.21.15.184:0/3062600162 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7590060420 con 0x7f75900f51a0 2024-02-19T22:17:22.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.660+0000 7f7577fff700 1 -- 172.21.15.184:0/3062600162 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7580013380 con 0x7f75900f51a0 2024-02-19T22:17:22.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.660+0000 7f7577fff700 1 -- 172.21.15.184:0/3062600162 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7580014e20 con 0x7f75900f51a0 2024-02-19T22:17:22.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.660+0000 7f7594a7d700 1 -- 172.21.15.184:0/3062600162 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7590109970 con 0x7f75900f51a0 2024-02-19T22:17:22.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.660+0000 7f7577fff700 1 -- 172.21.15.184:0/3062600162 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75800095f0 con 0x7f75900f51a0 2024-02-19T22:17:22.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.660+0000 7f7594a7d700 1 -- 172.21.15.184:0/3062600162 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7590059100 con 0x7f75900f51a0 2024-02-19T22:17:22.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.660+0000 7f7577fff700 1 --2- 172.21.15.184:0/3062600162 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f757802ff10 0x7f75780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:22.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.661+0000 7f7577fff700 1 -- 172.21.15.184:0/3062600162 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7580040f10 con 0x7f75900f51a0 2024-02-19T22:17:22.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.661+0000 7f758effd700 1 --2- 172.21.15.184:0/3062600162 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f757802ff10 0x7f75780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:22.664 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.661+0000 7f758effd700 1 --2- 172.21.15.184:0/3062600162 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f757802ff10 0x7f75780323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f75840096a0 tx=0x7f7584006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:22.667 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.665+0000 7f7577fff700 1 -- 172.21.15.184:0/3062600162 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7580014420 con 0x7f75900f51a0 2024-02-19T22:17:22.875 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.873+0000 7f7594a7d700 1 -- 172.21.15.184:0/3062600162 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7590045ad0 con 0x7f75900f51a0 2024-02-19T22:17:22.876 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.875+0000 7f7577fff700 1 -- 172.21.15.184:0/3062600162 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f758001d980 con 0x7f75900f51a0 2024-02-19T22:17:22.876 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:22.876 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:22.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.876+0000 7f7594a7d700 1 -- 172.21.15.184:0/3062600162 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f757802ff10 msgr2=0x7f75780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:22.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.876+0000 7f7594a7d700 1 --2- 172.21.15.184:0/3062600162 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f757802ff10 0x7f75780323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f75840096a0 tx=0x7f7584006b40).stop 2024-02-19T22:17:22.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.876+0000 7f7594a7d700 1 -- 172.21.15.184:0/3062600162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900f51a0 msgr2=0x7f75901092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:22.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.876+0000 7f7594a7d700 1 --2- 172.21.15.184:0/3062600162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900f51a0 0x7f75901092c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f75800089f0 tx=0x7f758000b040).stop 2024-02-19T22:17:22.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.876+0000 7f7594a7d700 1 -- 172.21.15.184:0/3062600162 shutdown_connections 2024-02-19T22:17:22.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.876+0000 7f7594a7d700 1 --2- 172.21.15.184:0/3062600162 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f757802ff10 0x7f75780323d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:22.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.876+0000 7f7594a7d700 1 --2- 172.21.15.184:0/3062600162 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f75900f51a0 0x7f75901092c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:22.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.876+0000 7f7594a7d700 1 -- 172.21.15.184:0/3062600162 >> 172.21.15.184:0/3062600162 conn(0x7f75900f09a0 msgr2=0x7f75900f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:22.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.876+0000 7f7594a7d700 1 -- 172.21.15.184:0/3062600162 shutdown_connections 2024-02-19T22:17:22.878 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:22.876+0000 7f7594a7d700 1 -- 172.21.15.184:0/3062600162 wait complete. 2024-02-19T22:17:22.879 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:24.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:21.911951+0000 mgr.smithi176.faoogw (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:24.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:22.875447+0000 mon.smithi176 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.184:0/3062600162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:24.761 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:24.761 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:26.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:23.912245+0000 mgr.smithi176.faoogw (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:26.668 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.666+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/1261544843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf00f5410 msgr2=0x7fbaf00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:26.669 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.666+0000 7fbaf6ff2700 1 --2- 172.21.15.184:0/1261544843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf00f5410 0x7fbaf00f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fbae4009fb0 tx=0x7fbae4009b30).stop 2024-02-19T22:17:26.669 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.666+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/1261544843 shutdown_connections 2024-02-19T22:17:26.669 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.666+0000 7fbaf6ff2700 1 --2- 172.21.15.184:0/1261544843 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf00f5410 0x7fbaf00f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:26.669 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.666+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/1261544843 >> 172.21.15.184:0/1261544843 conn(0x7fbaf00f0b70 msgr2=0x7fbaf00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:26.669 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.666+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/1261544843 shutdown_connections 2024-02-19T22:17:26.669 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.666+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/1261544843 wait complete. 2024-02-19T22:17:26.669 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.667+0000 7fbaf6ff2700 1 Processor -- start 2024-02-19T22:17:26.669 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.667+0000 7fbaf6ff2700 1 -- start start 2024-02-19T22:17:26.671 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.667+0000 7fbaf6ff2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf00f5410 0x7fbaf0109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:26.671 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.667+0000 7fbaf6ff2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf0060220 con 0x7fbaf00f5410 2024-02-19T22:17:26.671 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.667+0000 7fbaf4d8e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf00f5410 0x7fbaf0109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:26.671 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.667+0000 7fbaf4d8e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf00f5410 0x7fbaf0109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:36648/0 (socket says 172.21.15.184:36648) 2024-02-19T22:17:26.671 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.667+0000 7fbaf4d8e700 1 -- 172.21.15.184:0/2249031635 learned_addr learned my addr 172.21.15.184:0/2249031635 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:26.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.668+0000 7fbaf4d8e700 1 -- 172.21.15.184:0/2249031635 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae4005040 con 0x7fbaf00f5410 2024-02-19T22:17:26.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.668+0000 7fbaf4d8e700 1 --2- 172.21.15.184:0/2249031635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf00f5410 0x7fbaf0109500 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fbae4005010 tx=0x7fbae400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:26.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.668+0000 7fbaed7fa700 1 -- 172.21.15.184:0/2249031635 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbae4009320 con 0x7fbaf00f5410 2024-02-19T22:17:26.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.668+0000 7fbaed7fa700 1 -- 172.21.15.184:0/2249031635 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbae4013380 con 0x7fbaf00f5410 2024-02-19T22:17:26.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.668+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/2249031635 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf0060420 con 0x7fbaf00f5410 2024-02-19T22:17:26.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.668+0000 7fbaed7fa700 1 -- 172.21.15.184:0/2249031635 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbae4014e50 con 0x7fbaf00f5410 2024-02-19T22:17:26.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.668+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/2249031635 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf0109bb0 con 0x7fbaf00f5410 2024-02-19T22:17:26.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.669+0000 7fbaed7fa700 1 -- 172.21.15.184:0/2249031635 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbae40263a0 con 0x7fbaf00f5410 2024-02-19T22:17:26.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.669+0000 7fbaed7fa700 1 --2- 172.21.15.184:0/2249031635 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbae002ff10 0x7fbae00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:26.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.669+0000 7fbaed7fa700 1 -- 172.21.15.184:0/2249031635 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbae4040ea0 con 0x7fbaf00f5410 2024-02-19T22:17:26.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.669+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/2249031635 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf0059100 con 0x7fbaf00f5410 2024-02-19T22:17:26.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.669+0000 7fbaeffff700 1 --2- 172.21.15.184:0/2249031635 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbae002ff10 0x7fbae00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:26.672 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.669+0000 7fbaeffff700 1 --2- 172.21.15.184:0/2249031635 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbae002ff10 0x7fbae00323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fbadc0096a0 tx=0x7fbadc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:26.675 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.674+0000 7fbaed7fa700 1 -- 172.21.15.184:0/2249031635 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbae4012920 con 0x7fbaf00f5410 2024-02-19T22:17:26.879 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.878+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/2249031635 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbaf00fafa0 con 0x7fbaf00f5410 2024-02-19T22:17:26.880 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.878+0000 7fbaed7fa700 1 -- 172.21.15.184:0/2249031635 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbae4012330 con 0x7fbaf00f5410 2024-02-19T22:17:26.880 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:26.880 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:26.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.880+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/2249031635 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbae002ff10 msgr2=0x7fbae00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:26.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.880+0000 7fbaf6ff2700 1 --2- 172.21.15.184:0/2249031635 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbae002ff10 0x7fbae00323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fbadc0096a0 tx=0x7fbadc006b40).stop 2024-02-19T22:17:26.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.880+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/2249031635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf00f5410 msgr2=0x7fbaf0109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:26.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.880+0000 7fbaf6ff2700 1 --2- 172.21.15.184:0/2249031635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf00f5410 0x7fbaf0109500 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fbae4005010 tx=0x7fbae400b040).stop 2024-02-19T22:17:26.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.880+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/2249031635 shutdown_connections 2024-02-19T22:17:26.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.880+0000 7fbaf6ff2700 1 --2- 172.21.15.184:0/2249031635 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fbae002ff10 0x7fbae00323d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:26.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.880+0000 7fbaf6ff2700 1 --2- 172.21.15.184:0/2249031635 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbaf00f5410 0x7fbaf0109500 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:26.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.880+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/2249031635 >> 172.21.15.184:0/2249031635 conn(0x7fbaf00f0b70 msgr2=0x7fbaf00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:26.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.880+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/2249031635 shutdown_connections 2024-02-19T22:17:26.882 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:26.880+0000 7fbaf6ff2700 1 -- 172.21.15.184:0/2249031635 wait complete. 2024-02-19T22:17:26.883 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:28.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:25.912547+0000 mgr.smithi176.faoogw (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:28.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:26.879664+0000 mon.smithi176 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.184:0/2249031635' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:28.609 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:28.610 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:29.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:28 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:27.912846+0000 mgr.smithi176.faoogw (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:30.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.470+0000 7f78cf59e700 1 -- 172.21.15.184:0/1397080144 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78d00f62a0 msgr2=0x7f78d00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:30.477 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.470+0000 7f78cf59e700 1 --2- 172.21.15.184:0/1397080144 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78d00f62a0 0x7f78d00f66c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f78c0009fb0 tx=0x7f78c0009b30).stop 2024-02-19T22:17:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.471+0000 7f78cf59e700 1 -- 172.21.15.184:0/1397080144 shutdown_connections 2024-02-19T22:17:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.471+0000 7f78cf59e700 1 --2- 172.21.15.184:0/1397080144 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78d00f62a0 0x7f78d00f66c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.471+0000 7f78cf59e700 1 -- 172.21.15.184:0/1397080144 >> 172.21.15.184:0/1397080144 conn(0x7f78d00f1aa0 msgr2=0x7f78d00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.471+0000 7f78cf59e700 1 -- 172.21.15.184:0/1397080144 shutdown_connections 2024-02-19T22:17:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.471+0000 7f78cf59e700 1 -- 172.21.15.184:0/1397080144 wait complete. 2024-02-19T22:17:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.471+0000 7f78cf59e700 1 Processor -- start 2024-02-19T22:17:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.471+0000 7f78cf59e700 1 -- start start 2024-02-19T22:17:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.471+0000 7f78cf59e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78d00f62a0 0x7f78d006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.471+0000 7f78cf59e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0069e20 con 0x7f78d00f62a0 2024-02-19T22:17:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.471+0000 7f78ce59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78d00f62a0 0x7f78d006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.471+0000 7f78ce59c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78d00f62a0 0x7f78d006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:36678/0 (socket says 172.21.15.184:36678) 2024-02-19T22:17:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.471+0000 7f78ce59c700 1 -- 172.21.15.184:0/651834826 learned_addr learned my addr 172.21.15.184:0/651834826 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:30.478 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.472+0000 7f78ce59c700 1 -- 172.21.15.184:0/651834826 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c0005040 con 0x7f78d00f62a0 2024-02-19T22:17:30.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.472+0000 7f78ce59c700 1 --2- 172.21.15.184:0/651834826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78d00f62a0 0x7f78d006b9a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f78c00089f0 tx=0x7f78c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:30.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.472+0000 7f78beffd700 1 -- 172.21.15.184:0/651834826 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78c0009320 con 0x7f78d00f62a0 2024-02-19T22:17:30.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.472+0000 7f78beffd700 1 -- 172.21.15.184:0/651834826 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78c0013380 con 0x7f78d00f62a0 2024-02-19T22:17:30.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.472+0000 7f78cf59e700 1 -- 172.21.15.184:0/651834826 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d0069fc0 con 0x7f78d00f62a0 2024-02-19T22:17:30.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.472+0000 7f78beffd700 1 -- 172.21.15.184:0/651834826 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78c0014e20 con 0x7f78d00f62a0 2024-02-19T22:17:30.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.472+0000 7f78cf59e700 1 -- 172.21.15.184:0/651834826 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d006a450 con 0x7f78d00f62a0 2024-02-19T22:17:30.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.473+0000 7f78beffd700 1 -- 172.21.15.184:0/651834826 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78c0014420 con 0x7f78d00f62a0 2024-02-19T22:17:30.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.473+0000 7f78cf59e700 1 -- 172.21.15.184:0/651834826 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d0059100 con 0x7f78d00f62a0 2024-02-19T22:17:30.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.473+0000 7f78beffd700 1 --2- 172.21.15.184:0/651834826 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f78b802ff10 0x7f78b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:30.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.473+0000 7f78beffd700 1 -- 172.21.15.184:0/651834826 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78c0040fc0 con 0x7f78d00f62a0 2024-02-19T22:17:30.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.473+0000 7f78cdd9b700 1 --2- 172.21.15.184:0/651834826 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f78b802ff10 0x7f78b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:30.479 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.473+0000 7f78cdd9b700 1 --2- 172.21.15.184:0/651834826 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f78b802ff10 0x7f78b80323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f78c40096a0 tx=0x7f78c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:30.480 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.477+0000 7f78beffd700 1 -- 172.21.15.184:0/651834826 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78c006d2b0 con 0x7f78d00f62a0 2024-02-19T22:17:30.684 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.683+0000 7f78cf59e700 1 -- 172.21.15.184:0/651834826 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78d00fbe30 con 0x7f78d00f62a0 2024-02-19T22:17:30.685 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.683+0000 7f78beffd700 1 -- 172.21.15.184:0/651834826 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f78c00095f0 con 0x7f78d00f62a0 2024-02-19T22:17:30.685 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:30.686 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:30.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.685+0000 7f78cf59e700 1 -- 172.21.15.184:0/651834826 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f78b802ff10 msgr2=0x7f78b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:30.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.685+0000 7f78cf59e700 1 --2- 172.21.15.184:0/651834826 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f78b802ff10 0x7f78b80323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f78c40096a0 tx=0x7f78c4006b40).stop 2024-02-19T22:17:30.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.685+0000 7f78cf59e700 1 -- 172.21.15.184:0/651834826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78d00f62a0 msgr2=0x7f78d006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:30.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.685+0000 7f78cf59e700 1 --2- 172.21.15.184:0/651834826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78d00f62a0 0x7f78d006b9a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f78c00089f0 tx=0x7f78c000b040).stop 2024-02-19T22:17:30.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.685+0000 7f78cf59e700 1 -- 172.21.15.184:0/651834826 shutdown_connections 2024-02-19T22:17:30.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.685+0000 7f78cf59e700 1 --2- 172.21.15.184:0/651834826 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f78b802ff10 0x7f78b80323d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:30.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.685+0000 7f78cf59e700 1 --2- 172.21.15.184:0/651834826 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78d00f62a0 0x7f78d006b9a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:30.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.685+0000 7f78cf59e700 1 -- 172.21.15.184:0/651834826 >> 172.21.15.184:0/651834826 conn(0x7f78d00f1aa0 msgr2=0x7f78d00f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:30.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.685+0000 7f78cf59e700 1 -- 172.21.15.184:0/651834826 shutdown_connections 2024-02-19T22:17:30.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:30.685+0000 7f78cf59e700 1 -- 172.21.15.184:0/651834826 wait complete. 2024-02-19T22:17:30.688 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:31.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:30 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:30.684690+0000 mon.smithi176 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.184:0/651834826' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:32.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:29.913191+0000 mgr.smithi176.faoogw (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:32.378 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:32.378 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:34.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:31.913489+0000 mgr.smithi176.faoogw (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:34.342 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.339+0000 7f8463131700 1 -- 172.21.15.184:0/118192506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f845c0f5410 msgr2=0x7f845c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:34.342 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.339+0000 7f8463131700 1 --2- 172.21.15.184:0/118192506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f845c0f5410 0x7f845c0f5830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f8450009fb0 tx=0x7f8450009b30).stop 2024-02-19T22:17:34.342 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.339+0000 7f8463131700 1 -- 172.21.15.184:0/118192506 shutdown_connections 2024-02-19T22:17:34.342 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.339+0000 7f8463131700 1 --2- 172.21.15.184:0/118192506 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f845c0f5410 0x7f845c0f5830 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:34.342 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.339+0000 7f8463131700 1 -- 172.21.15.184:0/118192506 >> 172.21.15.184:0/118192506 conn(0x7f845c0f0b70 msgr2=0x7f845c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:34.342 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.339+0000 7f8463131700 1 -- 172.21.15.184:0/118192506 shutdown_connections 2024-02-19T22:17:34.342 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.339+0000 7f8463131700 1 -- 172.21.15.184:0/118192506 wait complete. 2024-02-19T22:17:34.342 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.340+0000 7f8463131700 1 Processor -- start 2024-02-19T22:17:34.343 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.340+0000 7f8463131700 1 -- start start 2024-02-19T22:17:34.345 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.340+0000 7f8463131700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f845c0f5410 0x7f845c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:34.345 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.340+0000 7f8463131700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c060240 con 0x7f845c0f5410 2024-02-19T22:17:34.345 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.340+0000 7f8460ecd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f845c0f5410 0x7f845c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:34.345 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.340+0000 7f8460ecd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f845c0f5410 0x7f845c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:36696/0 (socket says 172.21.15.184:36696) 2024-02-19T22:17:34.345 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.340+0000 7f8460ecd700 1 -- 172.21.15.184:0/957133280 learned_addr learned my addr 172.21.15.184:0/957133280 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:34.345 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.341+0000 7f8460ecd700 1 -- 172.21.15.184:0/957133280 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8450005040 con 0x7f845c0f5410 2024-02-19T22:17:34.345 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.341+0000 7f8460ecd700 1 --2- 172.21.15.184:0/957133280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f845c0f5410 0x7f845c1094e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f8450005010 tx=0x7f845000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:34.345 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.341+0000 7f84597fa700 1 -- 172.21.15.184:0/957133280 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8450009320 con 0x7f845c0f5410 2024-02-19T22:17:34.345 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.341+0000 7f8463131700 1 -- 172.21.15.184:0/957133280 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845c060440 con 0x7f845c0f5410 2024-02-19T22:17:34.345 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.341+0000 7f84597fa700 1 -- 172.21.15.184:0/957133280 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8450013380 con 0x7f845c0f5410 2024-02-19T22:17:34.345 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.341+0000 7f84597fa700 1 -- 172.21.15.184:0/957133280 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8450014e50 con 0x7f845c0f5410 2024-02-19T22:17:34.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.341+0000 7f8463131700 1 -- 172.21.15.184:0/957133280 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845c109b90 con 0x7f845c0f5410 2024-02-19T22:17:34.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.342+0000 7f84597fa700 1 -- 172.21.15.184:0/957133280 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84500095f0 con 0x7f845c0f5410 2024-02-19T22:17:34.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.342+0000 7f84597fa700 1 --2- 172.21.15.184:0/957133280 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f844c02ff10 0x7f844c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:34.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.342+0000 7f84597fa700 1 -- 172.21.15.184:0/957133280 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8450040e50 con 0x7f845c0f5410 2024-02-19T22:17:34.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.342+0000 7f8463131700 1 -- 172.21.15.184:0/957133280 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845c045ad0 con 0x7f845c0f5410 2024-02-19T22:17:34.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.342+0000 7f845bfff700 1 --2- 172.21.15.184:0/957133280 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f844c02ff10 0x7f844c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:34.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.343+0000 7f845bfff700 1 --2- 172.21.15.184:0/957133280 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f844c02ff10 0x7f844c0323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f84480096a0 tx=0x7f8448006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:34.348 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.347+0000 7f84597fa700 1 -- 172.21.15.184:0/957133280 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8450014420 con 0x7f845c0f5410 2024-02-19T22:17:34.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.551+0000 7f8463131700 1 -- 172.21.15.184:0/957133280 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f845c10a270 con 0x7f845c0f5410 2024-02-19T22:17:34.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.552+0000 7f84597fa700 1 -- 172.21.15.184:0/957133280 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8450014420 con 0x7f845c0f5410 2024-02-19T22:17:34.554 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:34.554 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:34.555 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.553+0000 7f8463131700 1 -- 172.21.15.184:0/957133280 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f844c02ff10 msgr2=0x7f844c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:34.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.553+0000 7f8463131700 1 --2- 172.21.15.184:0/957133280 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f844c02ff10 0x7f844c0323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f84480096a0 tx=0x7f8448006b40).stop 2024-02-19T22:17:34.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.554+0000 7f8463131700 1 -- 172.21.15.184:0/957133280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f845c0f5410 msgr2=0x7f845c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:34.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.554+0000 7f8463131700 1 --2- 172.21.15.184:0/957133280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f845c0f5410 0x7f845c1094e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f8450005010 tx=0x7f845000b040).stop 2024-02-19T22:17:34.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.554+0000 7f8463131700 1 -- 172.21.15.184:0/957133280 shutdown_connections 2024-02-19T22:17:34.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.554+0000 7f8463131700 1 --2- 172.21.15.184:0/957133280 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f844c02ff10 0x7f844c0323d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:34.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.554+0000 7f8463131700 1 --2- 172.21.15.184:0/957133280 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f845c0f5410 0x7f845c1094e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:34.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.554+0000 7f8463131700 1 -- 172.21.15.184:0/957133280 >> 172.21.15.184:0/957133280 conn(0x7f845c0f0b70 msgr2=0x7f845c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:34.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.554+0000 7f8463131700 1 -- 172.21.15.184:0/957133280 shutdown_connections 2024-02-19T22:17:34.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:34.554+0000 7f8463131700 1 -- 172.21.15.184:0/957133280 wait complete. 2024-02-19T22:17:34.557 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:35.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:34 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:34.553225+0000 mon.smithi176 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.184:0/957133280' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:36.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:33.913817+0000 mgr.smithi176.faoogw (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:36.291 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:36.291 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:35.974905+0000 mon.smithi176 (mon.0) 276 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:17:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:35.976441+0000 mon.smithi176 (mon.0) 277 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:17:38.103 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.095+0000 7fe686f0a700 1 -- 172.21.15.184:0/3922445647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6800f5410 msgr2=0x7fe6800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:38.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.095+0000 7fe686f0a700 1 --2- 172.21.15.184:0/3922445647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6800f5410 0x7fe6800f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fe674009fb0 tx=0x7fe674009b30).stop 2024-02-19T22:17:38.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.095+0000 7fe686f0a700 1 -- 172.21.15.184:0/3922445647 shutdown_connections 2024-02-19T22:17:38.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.095+0000 7fe686f0a700 1 --2- 172.21.15.184:0/3922445647 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6800f5410 0x7fe6800f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:38.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.095+0000 7fe686f0a700 1 -- 172.21.15.184:0/3922445647 >> 172.21.15.184:0/3922445647 conn(0x7fe6800f0b70 msgr2=0x7fe6800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:38.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.095+0000 7fe686f0a700 1 -- 172.21.15.184:0/3922445647 shutdown_connections 2024-02-19T22:17:38.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.095+0000 7fe686f0a700 1 -- 172.21.15.184:0/3922445647 wait complete. 2024-02-19T22:17:38.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.096+0000 7fe686f0a700 1 Processor -- start 2024-02-19T22:17:38.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.096+0000 7fe686f0a700 1 -- start start 2024-02-19T22:17:38.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.096+0000 7fe686f0a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6800f5410 0x7fe6801094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:38.104 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.096+0000 7fe686f0a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe680060240 con 0x7fe6800f5410 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.096+0000 7fe684ca6700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6800f5410 0x7fe6801094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.096+0000 7fe684ca6700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6800f5410 0x7fe6801094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:42908/0 (socket says 172.21.15.184:42908) 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.096+0000 7fe684ca6700 1 -- 172.21.15.184:0/3970907702 learned_addr learned my addr 172.21.15.184:0/3970907702 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.096+0000 7fe684ca6700 1 -- 172.21.15.184:0/3970907702 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe674005040 con 0x7fe6800f5410 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.096+0000 7fe684ca6700 1 --2- 172.21.15.184:0/3970907702 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6800f5410 0x7fe6801094e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fe674005010 tx=0x7fe67400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.097+0000 7fe67d7fa700 1 -- 172.21.15.184:0/3970907702 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe674009320 con 0x7fe6800f5410 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.097+0000 7fe686f0a700 1 -- 172.21.15.184:0/3970907702 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe680060440 con 0x7fe6800f5410 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.097+0000 7fe67d7fa700 1 -- 172.21.15.184:0/3970907702 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe674013380 con 0x7fe6800f5410 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.097+0000 7fe67d7fa700 1 -- 172.21.15.184:0/3970907702 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe674014e50 con 0x7fe6800f5410 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.097+0000 7fe686f0a700 1 -- 172.21.15.184:0/3970907702 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe680109b90 con 0x7fe6800f5410 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.097+0000 7fe67d7fa700 1 -- 172.21.15.184:0/3970907702 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe6740095f0 con 0x7fe6800f5410 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.097+0000 7fe686f0a700 1 -- 172.21.15.184:0/3970907702 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe680045ad0 con 0x7fe6800f5410 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.098+0000 7fe67d7fa700 1 --2- 172.21.15.184:0/3970907702 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe67002ff10 0x7fe6700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.098+0000 7fe67d7fa700 1 -- 172.21.15.184:0/3970907702 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe674040e90 con 0x7fe6800f5410 2024-02-19T22:17:38.105 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.098+0000 7fe67ffff700 1 --2- 172.21.15.184:0/3970907702 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe67002ff10 0x7fe6700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:38.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.098+0000 7fe67ffff700 1 --2- 172.21.15.184:0/3970907702 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe67002ff10 0x7fe6700323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fe66c0096a0 tx=0x7fe66c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:38.106 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.102+0000 7fe67d7fa700 1 -- 172.21.15.184:0/3970907702 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe674014420 con 0x7fe6800f5410 2024-02-19T22:17:38.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:35.914182+0000 mgr.smithi176.faoogw (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:38.308 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.307+0000 7fe686f0a700 1 -- 172.21.15.184:0/3970907702 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe68010a3b0 con 0x7fe6800f5410 2024-02-19T22:17:38.308 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.308+0000 7fe67d7fa700 1 -- 172.21.15.184:0/3970907702 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe674010230 con 0x7fe6800f5410 2024-02-19T22:17:38.309 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:38.309 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:38.310 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.309+0000 7fe686f0a700 1 -- 172.21.15.184:0/3970907702 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe67002ff10 msgr2=0x7fe6700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:38.310 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.309+0000 7fe686f0a700 1 --2- 172.21.15.184:0/3970907702 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe67002ff10 0x7fe6700323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fe66c0096a0 tx=0x7fe66c006b40).stop 2024-02-19T22:17:38.310 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.309+0000 7fe686f0a700 1 -- 172.21.15.184:0/3970907702 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6800f5410 msgr2=0x7fe6801094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:38.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.309+0000 7fe686f0a700 1 --2- 172.21.15.184:0/3970907702 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6800f5410 0x7fe6801094e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fe674005010 tx=0x7fe67400b040).stop 2024-02-19T22:17:38.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.309+0000 7fe686f0a700 1 -- 172.21.15.184:0/3970907702 shutdown_connections 2024-02-19T22:17:38.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.309+0000 7fe686f0a700 1 --2- 172.21.15.184:0/3970907702 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fe67002ff10 0x7fe6700323d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:38.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.309+0000 7fe686f0a700 1 --2- 172.21.15.184:0/3970907702 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe6800f5410 0x7fe6801094e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:38.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.309+0000 7fe686f0a700 1 -- 172.21.15.184:0/3970907702 >> 172.21.15.184:0/3970907702 conn(0x7fe6800f0b70 msgr2=0x7fe6800f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:38.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.310+0000 7fe686f0a700 1 -- 172.21.15.184:0/3970907702 shutdown_connections 2024-02-19T22:17:38.311 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:38.310+0000 7fe686f0a700 1 -- 172.21.15.184:0/3970907702 wait complete. 2024-02-19T22:17:38.311 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:39.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:38 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:38.307986+0000 mon.smithi176 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.184:0/3970907702' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:40.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:40.050 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:40.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:37.914509+0000 mgr.smithi176.faoogw (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:42.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.058+0000 7f14a5506700 1 -- 172.21.15.184:0/2772183697 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14a00f6510 msgr2=0x7f14a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:42.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.058+0000 7f14a5506700 1 --2- 172.21.15.184:0/2772183697 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14a00f6510 0x7f14a00f6930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f1490009fb0 tx=0x7f1490009b30).stop 2024-02-19T22:17:42.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.058+0000 7f14a5506700 1 -- 172.21.15.184:0/2772183697 shutdown_connections 2024-02-19T22:17:42.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.058+0000 7f14a5506700 1 --2- 172.21.15.184:0/2772183697 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14a00f6510 0x7f14a00f6930 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:42.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.058+0000 7f14a5506700 1 -- 172.21.15.184:0/2772183697 >> 172.21.15.184:0/2772183697 conn(0x7f14a00f1c70 msgr2=0x7f14a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:42.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.058+0000 7f14a5506700 1 -- 172.21.15.184:0/2772183697 shutdown_connections 2024-02-19T22:17:42.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.058+0000 7f14a5506700 1 -- 172.21.15.184:0/2772183697 wait complete. 2024-02-19T22:17:42.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.059+0000 7f14a5506700 1 Processor -- start 2024-02-19T22:17:42.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.059+0000 7f14a5506700 1 -- start start 2024-02-19T22:17:42.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.059+0000 7f14a5506700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14a0106900 0x7f14a0106d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:42.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.059+0000 7f14a5506700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a0107210 con 0x7f14a0106900 2024-02-19T22:17:42.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.059+0000 7f149effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14a0106900 0x7f14a0106d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:42.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.059+0000 7f149effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14a0106900 0x7f14a0106d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:42930/0 (socket says 172.21.15.184:42930) 2024-02-19T22:17:42.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.059+0000 7f149effd700 1 -- 172.21.15.184:0/475503428 learned_addr learned my addr 172.21.15.184:0/475503428 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:42.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.060+0000 7f149effd700 1 -- 172.21.15.184:0/475503428 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1490005040 con 0x7f14a0106900 2024-02-19T22:17:42.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.060+0000 7f149effd700 1 --2- 172.21.15.184:0/475503428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14a0106900 0x7f14a0106d20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f1490009f80 tx=0x7f1490009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:42.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.060+0000 7f14877fe700 1 -- 172.21.15.184:0/475503428 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1490013380 con 0x7f14a0106900 2024-02-19T22:17:42.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.060+0000 7f14a5506700 1 -- 172.21.15.184:0/475503428 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a0107410 con 0x7f14a0106900 2024-02-19T22:17:42.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.060+0000 7f14877fe700 1 -- 172.21.15.184:0/475503428 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1490013790 con 0x7f14a0106900 2024-02-19T22:17:42.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.060+0000 7f14877fe700 1 -- 172.21.15.184:0/475503428 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1490014420 con 0x7f14a0106900 2024-02-19T22:17:42.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.060+0000 7f14a5506700 1 -- 172.21.15.184:0/475503428 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a0107f60 con 0x7f14a0106900 2024-02-19T22:17:42.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.061+0000 7f14877fe700 1 -- 172.21.15.184:0/475503428 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1490025060 con 0x7f14a0106900 2024-02-19T22:17:42.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.061+0000 7f14877fe700 1 --2- 172.21.15.184:0/475503428 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f148802ff60 0x7f1488032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:42.067 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.061+0000 7f14877fe700 1 -- 172.21.15.184:0/475503428 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1490040da0 con 0x7f14a0106900 2024-02-19T22:17:42.067 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.061+0000 7f14a5506700 1 -- 172.21.15.184:0/475503428 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a0059100 con 0x7f14a0106900 2024-02-19T22:17:42.067 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.061+0000 7f149e7fc700 1 --2- 172.21.15.184:0/475503428 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f148802ff60 0x7f1488032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:42.067 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.062+0000 7f149e7fc700 1 --2- 172.21.15.184:0/475503428 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f148802ff60 0x7f1488032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f14940096a0 tx=0x7f1494006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:42.068 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.066+0000 7f14877fe700 1 -- 172.21.15.184:0/475503428 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1490028440 con 0x7f14a0106900 2024-02-19T22:17:42.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:39.914866+0000 mgr.smithi176.faoogw (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:42.272 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.270+0000 7f14a5506700 1 -- 172.21.15.184:0/475503428 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14a01085c0 con 0x7f14a0106900 2024-02-19T22:17:42.272 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.270+0000 7f14877fe700 1 -- 172.21.15.184:0/475503428 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1490026340 con 0x7f14a0106900 2024-02-19T22:17:42.273 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:42.273 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:42.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.272+0000 7f14a5506700 1 -- 172.21.15.184:0/475503428 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f148802ff60 msgr2=0x7f1488032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:42.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.272+0000 7f14a5506700 1 --2- 172.21.15.184:0/475503428 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f148802ff60 0x7f1488032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f14940096a0 tx=0x7f1494006b40).stop 2024-02-19T22:17:42.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.272+0000 7f14a5506700 1 -- 172.21.15.184:0/475503428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14a0106900 msgr2=0x7f14a0106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:42.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.272+0000 7f14a5506700 1 --2- 172.21.15.184:0/475503428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14a0106900 0x7f14a0106d20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f1490009f80 tx=0x7f1490009b30).stop 2024-02-19T22:17:42.274 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.272+0000 7f14a5506700 1 -- 172.21.15.184:0/475503428 shutdown_connections 2024-02-19T22:17:42.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.272+0000 7f14a5506700 1 --2- 172.21.15.184:0/475503428 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f148802ff60 0x7f1488032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:42.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.272+0000 7f14a5506700 1 --2- 172.21.15.184:0/475503428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14a0106900 0x7f14a0106d20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:42.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.272+0000 7f14a5506700 1 -- 172.21.15.184:0/475503428 >> 172.21.15.184:0/475503428 conn(0x7f14a00f1c70 msgr2=0x7f14a00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:42.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.272+0000 7f14a5506700 1 -- 172.21.15.184:0/475503428 shutdown_connections 2024-02-19T22:17:42.275 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:42.272+0000 7f14a5506700 1 -- 172.21.15.184:0/475503428 wait complete. 2024-02-19T22:17:42.275 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:43.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:42 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:42.271880+0000 mon.smithi176 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.184:0/475503428' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:43.969 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:43.969 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:44.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:41.915239+0000 mgr.smithi176.faoogw (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:45.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.930+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/2107476912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2e80f6510 msgr2=0x7fd2e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:45.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.930+0000 7fd2ecbdc700 1 --2- 172.21.15.184:0/2107476912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2e80f6510 0x7fd2e80f6930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fd2d8009fb0 tx=0x7fd2d8009b30).stop 2024-02-19T22:17:45.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.930+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/2107476912 shutdown_connections 2024-02-19T22:17:45.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.930+0000 7fd2ecbdc700 1 --2- 172.21.15.184:0/2107476912 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2e80f6510 0x7fd2e80f6930 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:45.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.930+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/2107476912 >> 172.21.15.184:0/2107476912 conn(0x7fd2e80f1c70 msgr2=0x7fd2e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:45.933 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.930+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/2107476912 shutdown_connections 2024-02-19T22:17:45.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.930+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/2107476912 wait complete. 2024-02-19T22:17:45.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.931+0000 7fd2ecbdc700 1 Processor -- start 2024-02-19T22:17:45.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.931+0000 7fd2ecbdc700 1 -- start start 2024-02-19T22:17:45.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.931+0000 7fd2ecbdc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2e80f6510 0x7fd2e81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:45.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.931+0000 7fd2ecbdc700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e8106cd0 con 0x7fd2e80f6510 2024-02-19T22:17:45.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.931+0000 7fd2e659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2e80f6510 0x7fd2e81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:45.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.931+0000 7fd2e659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2e80f6510 0x7fd2e81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:42962/0 (socket says 172.21.15.184:42962) 2024-02-19T22:17:45.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.931+0000 7fd2e659c700 1 -- 172.21.15.184:0/3814519059 learned_addr learned my addr 172.21.15.184:0/3814519059 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:45.935 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.931+0000 7fd2e659c700 1 -- 172.21.15.184:0/3814519059 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d8005040 con 0x7fd2e80f6510 2024-02-19T22:17:45.935 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.932+0000 7fd2e659c700 1 --2- 172.21.15.184:0/3814519059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2e80f6510 0x7fd2e81067e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fd2d80089f0 tx=0x7fd2d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:45.935 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.932+0000 7fd2d6ffd700 1 -- 172.21.15.184:0/3814519059 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2d8009320 con 0x7fd2e80f6510 2024-02-19T22:17:45.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.932+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/3814519059 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e8106ed0 con 0x7fd2e80f6510 2024-02-19T22:17:45.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.932+0000 7fd2d6ffd700 1 -- 172.21.15.184:0/3814519059 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd2d8013380 con 0x7fd2e80f6510 2024-02-19T22:17:45.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.932+0000 7fd2d6ffd700 1 -- 172.21.15.184:0/3814519059 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2d801ed80 con 0x7fd2e80f6510 2024-02-19T22:17:45.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.932+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/3814519059 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e8107360 con 0x7fd2e80f6510 2024-02-19T22:17:45.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.933+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/3814519059 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e8045ad0 con 0x7fd2e80f6510 2024-02-19T22:17:45.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.933+0000 7fd2d6ffd700 1 -- 172.21.15.184:0/3814519059 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd2d801e420 con 0x7fd2e80f6510 2024-02-19T22:17:45.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.933+0000 7fd2d6ffd700 1 --2- 172.21.15.184:0/3814519059 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd2d002ff60 0x7fd2d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:45.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.933+0000 7fd2d6ffd700 1 -- 172.21.15.184:0/3814519059 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd2d8040f20 con 0x7fd2e80f6510 2024-02-19T22:17:45.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.933+0000 7fd2e5d9b700 1 --2- 172.21.15.184:0/3814519059 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd2d002ff60 0x7fd2d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:45.936 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.933+0000 7fd2e5d9b700 1 --2- 172.21.15.184:0/3814519059 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd2d002ff60 0x7fd2d0032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fd2dc0096a0 tx=0x7fd2dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:45.939 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:45.937+0000 7fd2d6ffd700 1 -- 172.21.15.184:0/3814519059 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2d80095f0 con 0x7fd2e80f6510 2024-02-19T22:17:46.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:43.915617+0000 mgr.smithi176.faoogw (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:46.144 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:46.142+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/3814519059 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd2e8108210 con 0x7fd2e80f6510 2024-02-19T22:17:46.144 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:46.143+0000 7fd2d6ffd700 1 -- 172.21.15.184:0/3814519059 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd2d801cb10 con 0x7fd2e80f6510 2024-02-19T22:17:46.145 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:46.145 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:46.146 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:46.144+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/3814519059 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd2d002ff60 msgr2=0x7fd2d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:46.146 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:46.144+0000 7fd2ecbdc700 1 --2- 172.21.15.184:0/3814519059 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd2d002ff60 0x7fd2d0032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fd2dc0096a0 tx=0x7fd2dc006b40).stop 2024-02-19T22:17:46.146 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:46.144+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/3814519059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2e80f6510 msgr2=0x7fd2e81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:46.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:46.144+0000 7fd2ecbdc700 1 --2- 172.21.15.184:0/3814519059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2e80f6510 0x7fd2e81067e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fd2d80089f0 tx=0x7fd2d800b040).stop 2024-02-19T22:17:46.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:46.144+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/3814519059 shutdown_connections 2024-02-19T22:17:46.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:46.144+0000 7fd2ecbdc700 1 --2- 172.21.15.184:0/3814519059 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd2d002ff60 0x7fd2d0032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:46.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:46.144+0000 7fd2ecbdc700 1 --2- 172.21.15.184:0/3814519059 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2e80f6510 0x7fd2e81067e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:46.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:46.144+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/3814519059 >> 172.21.15.184:0/3814519059 conn(0x7fd2e80f1c70 msgr2=0x7fd2e80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:46.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:46.144+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/3814519059 shutdown_connections 2024-02-19T22:17:46.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:46.144+0000 7fd2ecbdc700 1 -- 172.21.15.184:0/3814519059 wait complete. 2024-02-19T22:17:46.147 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:47.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:46 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:46.144076+0000 mon.smithi176 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.184:0/3814519059' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:47.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:47.763 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:48.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:45.915984+0000 mgr.smithi176.faoogw (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:49.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.689+0000 7fdd37142700 1 -- 172.21.15.184:0/2721101727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd300f6510 msgr2=0x7fdd300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:49.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.689+0000 7fdd37142700 1 --2- 172.21.15.184:0/2721101727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd300f6510 0x7fdd300f6930 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fdd24009fb0 tx=0x7fdd24009b30).stop 2024-02-19T22:17:49.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.689+0000 7fdd37142700 1 -- 172.21.15.184:0/2721101727 shutdown_connections 2024-02-19T22:17:49.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.689+0000 7fdd37142700 1 --2- 172.21.15.184:0/2721101727 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd300f6510 0x7fdd300f6930 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:49.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.689+0000 7fdd37142700 1 -- 172.21.15.184:0/2721101727 >> 172.21.15.184:0/2721101727 conn(0x7fdd300f1c70 msgr2=0x7fdd300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:49.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.689+0000 7fdd37142700 1 -- 172.21.15.184:0/2721101727 shutdown_connections 2024-02-19T22:17:49.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.689+0000 7fdd37142700 1 -- 172.21.15.184:0/2721101727 wait complete. 2024-02-19T22:17:49.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.690+0000 7fdd37142700 1 Processor -- start 2024-02-19T22:17:49.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.690+0000 7fdd37142700 1 -- start start 2024-02-19T22:17:49.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.690+0000 7fdd37142700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd300f6510 0x7fdd301067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:49.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.690+0000 7fdd37142700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd30106cd0 con 0x7fdd300f6510 2024-02-19T22:17:49.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.691+0000 7fdd34ede700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd300f6510 0x7fdd301067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:49.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.691+0000 7fdd34ede700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd300f6510 0x7fdd301067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:46356/0 (socket says 172.21.15.184:46356) 2024-02-19T22:17:49.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.691+0000 7fdd34ede700 1 -- 172.21.15.184:0/1763549033 learned_addr learned my addr 172.21.15.184:0/1763549033 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:49.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.691+0000 7fdd34ede700 1 -- 172.21.15.184:0/1763549033 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd24005040 con 0x7fdd300f6510 2024-02-19T22:17:49.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.691+0000 7fdd34ede700 1 --2- 172.21.15.184:0/1763549033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd300f6510 0x7fdd301067e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fdd24005010 tx=0x7fdd2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:49.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.691+0000 7fdd2d7fa700 1 -- 172.21.15.184:0/1763549033 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd24009320 con 0x7fdd300f6510 2024-02-19T22:17:49.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.691+0000 7fdd37142700 1 -- 172.21.15.184:0/1763549033 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd30106ed0 con 0x7fdd300f6510 2024-02-19T22:17:49.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.691+0000 7fdd2d7fa700 1 -- 172.21.15.184:0/1763549033 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd24013380 con 0x7fdd300f6510 2024-02-19T22:17:49.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.691+0000 7fdd2d7fa700 1 -- 172.21.15.184:0/1763549033 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd24014e50 con 0x7fdd300f6510 2024-02-19T22:17:49.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.692+0000 7fdd37142700 1 -- 172.21.15.184:0/1763549033 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd30107360 con 0x7fdd300f6510 2024-02-19T22:17:49.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.692+0000 7fdd2d7fa700 1 -- 172.21.15.184:0/1763549033 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd240095f0 con 0x7fdd300f6510 2024-02-19T22:17:49.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.692+0000 7fdd37142700 1 -- 172.21.15.184:0/1763549033 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd30045ad0 con 0x7fdd300f6510 2024-02-19T22:17:49.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.692+0000 7fdd2d7fa700 1 --2- 172.21.15.184:0/1763549033 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fdd2002ff10 0x7fdd200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:49.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.692+0000 7fdd2d7fa700 1 -- 172.21.15.184:0/1763549033 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd24040e90 con 0x7fdd300f6510 2024-02-19T22:17:49.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.693+0000 7fdd2ffff700 1 --2- 172.21.15.184:0/1763549033 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fdd2002ff10 0x7fdd200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:49.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.693+0000 7fdd2ffff700 1 --2- 172.21.15.184:0/1763549033 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fdd2002ff10 0x7fdd200323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fdd1c0096a0 tx=0x7fdd1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:49.699 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.697+0000 7fdd2d7fa700 1 -- 172.21.15.184:0/1763549033 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd24014420 con 0x7fdd300f6510 2024-02-19T22:17:49.908 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.906+0000 7fdd37142700 1 -- 172.21.15.184:0/1763549033 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd30108210 con 0x7fdd300f6510 2024-02-19T22:17:49.908 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.907+0000 7fdd2d7fa700 1 -- 172.21.15.184:0/1763549033 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdd24010230 con 0x7fdd300f6510 2024-02-19T22:17:49.909 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:49.909 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:49.910 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.908+0000 7fdd37142700 1 -- 172.21.15.184:0/1763549033 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fdd2002ff10 msgr2=0x7fdd200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:49.910 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.908+0000 7fdd37142700 1 --2- 172.21.15.184:0/1763549033 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fdd2002ff10 0x7fdd200323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fdd1c0096a0 tx=0x7fdd1c006b40).stop 2024-02-19T22:17:49.910 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.908+0000 7fdd37142700 1 -- 172.21.15.184:0/1763549033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd300f6510 msgr2=0x7fdd301067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:49.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.908+0000 7fdd37142700 1 --2- 172.21.15.184:0/1763549033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd300f6510 0x7fdd301067e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fdd24005010 tx=0x7fdd2400b040).stop 2024-02-19T22:17:49.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.908+0000 7fdd37142700 1 -- 172.21.15.184:0/1763549033 shutdown_connections 2024-02-19T22:17:49.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.908+0000 7fdd37142700 1 --2- 172.21.15.184:0/1763549033 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fdd2002ff10 0x7fdd200323d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:49.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.908+0000 7fdd37142700 1 --2- 172.21.15.184:0/1763549033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd300f6510 0x7fdd301067e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:49.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.908+0000 7fdd37142700 1 -- 172.21.15.184:0/1763549033 >> 172.21.15.184:0/1763549033 conn(0x7fdd300f1c70 msgr2=0x7fdd300f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:49.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.908+0000 7fdd37142700 1 -- 172.21.15.184:0/1763549033 shutdown_connections 2024-02-19T22:17:49.911 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:49.908+0000 7fdd37142700 1 -- 172.21.15.184:0/1763549033 wait complete. 2024-02-19T22:17:49.911 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:50.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:49 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:47.916368+0000 mgr.smithi176.faoogw (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:51.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:50 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:49.908145+0000 mon.smithi176 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.184:0/1763549033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:51.607 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:51.608 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:52.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:51 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:49.916749+0000 mgr.smithi176.faoogw (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:53.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.573+0000 7fab97fb1700 1 -- 172.21.15.184:0/2570506116 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab90060280 msgr2=0x7fab900f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:53.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.573+0000 7fab97fb1700 1 --2- 172.21.15.184:0/2570506116 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab90060280 0x7fab900f90f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fab80009fb0 tx=0x7fab80009b30).stop 2024-02-19T22:17:53.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.573+0000 7fab97fb1700 1 -- 172.21.15.184:0/2570506116 shutdown_connections 2024-02-19T22:17:53.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.573+0000 7fab97fb1700 1 --2- 172.21.15.184:0/2570506116 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab90060280 0x7fab900f90f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:53.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.573+0000 7fab97fb1700 1 -- 172.21.15.184:0/2570506116 >> 172.21.15.184:0/2570506116 conn(0x7fab900f0b70 msgr2=0x7fab900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:53.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.573+0000 7fab97fb1700 1 -- 172.21.15.184:0/2570506116 shutdown_connections 2024-02-19T22:17:53.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.573+0000 7fab97fb1700 1 -- 172.21.15.184:0/2570506116 wait complete. 2024-02-19T22:17:53.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.574+0000 7fab97fb1700 1 Processor -- start 2024-02-19T22:17:53.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.574+0000 7fab97fb1700 1 -- start start 2024-02-19T22:17:53.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.574+0000 7fab97fb1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab90060280 0x7fab90106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:53.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.574+0000 7fab97fb1700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab90106c70 con 0x7fab90060280 2024-02-19T22:17:53.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.574+0000 7fab95d4d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab90060280 0x7fab90106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:53.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.574+0000 7fab95d4d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab90060280 0x7fab90106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:46380/0 (socket says 172.21.15.184:46380) 2024-02-19T22:17:53.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.574+0000 7fab95d4d700 1 -- 172.21.15.184:0/1493029859 learned_addr learned my addr 172.21.15.184:0/1493029859 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:53.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.574+0000 7fab95d4d700 1 -- 172.21.15.184:0/1493029859 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab80005040 con 0x7fab90060280 2024-02-19T22:17:53.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.574+0000 7fab95d4d700 1 --2- 172.21.15.184:0/1493029859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab90060280 0x7fab90106780 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fab800089f0 tx=0x7fab8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:53.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.575+0000 7fab867fc700 1 -- 172.21.15.184:0/1493029859 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab80009320 con 0x7fab90060280 2024-02-19T22:17:53.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.575+0000 7fab867fc700 1 -- 172.21.15.184:0/1493029859 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab80013380 con 0x7fab90060280 2024-02-19T22:17:53.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.575+0000 7fab97fb1700 1 -- 172.21.15.184:0/1493029859 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab90106e70 con 0x7fab90060280 2024-02-19T22:17:53.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.575+0000 7fab867fc700 1 -- 172.21.15.184:0/1493029859 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab80014e20 con 0x7fab90060280 2024-02-19T22:17:53.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.575+0000 7fab97fb1700 1 -- 172.21.15.184:0/1493029859 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab90107300 con 0x7fab90060280 2024-02-19T22:17:53.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.575+0000 7fab867fc700 1 -- 172.21.15.184:0/1493029859 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab800095f0 con 0x7fab90060280 2024-02-19T22:17:53.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.575+0000 7fab97fb1700 1 -- 172.21.15.184:0/1493029859 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab90045ad0 con 0x7fab90060280 2024-02-19T22:17:53.590 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.576+0000 7fab867fc700 1 --2- 172.21.15.184:0/1493029859 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fab7c02ff10 0x7fab7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:53.590 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.576+0000 7fab867fc700 1 -- 172.21.15.184:0/1493029859 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab80040f50 con 0x7fab90060280 2024-02-19T22:17:53.590 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.576+0000 7fab9554c700 1 --2- 172.21.15.184:0/1493029859 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fab7c02ff10 0x7fab7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:53.590 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.576+0000 7fab9554c700 1 --2- 172.21.15.184:0/1493029859 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fab7c02ff10 0x7fab7c0323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fab8c0096a0 tx=0x7fab8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:53.590 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.580+0000 7fab867fc700 1 -- 172.21.15.184:0/1493029859 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab80014420 con 0x7fab90060280 2024-02-19T22:17:53.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.786+0000 7fab97fb1700 1 -- 172.21.15.184:0/1493029859 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab901081b0 con 0x7fab90060280 2024-02-19T22:17:53.788 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.786+0000 7fab867fc700 1 -- 172.21.15.184:0/1493029859 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fab8001d980 con 0x7fab90060280 2024-02-19T22:17:53.789 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:53.789 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:53.790 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.788+0000 7fab97fb1700 1 -- 172.21.15.184:0/1493029859 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fab7c02ff10 msgr2=0x7fab7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:53.790 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.788+0000 7fab97fb1700 1 --2- 172.21.15.184:0/1493029859 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fab7c02ff10 0x7fab7c0323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fab8c0096a0 tx=0x7fab8c006b40).stop 2024-02-19T22:17:53.790 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.788+0000 7fab97fb1700 1 -- 172.21.15.184:0/1493029859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab90060280 msgr2=0x7fab90106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:53.790 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.788+0000 7fab97fb1700 1 --2- 172.21.15.184:0/1493029859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab90060280 0x7fab90106780 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fab800089f0 tx=0x7fab8000b040).stop 2024-02-19T22:17:53.791 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.788+0000 7fab97fb1700 1 -- 172.21.15.184:0/1493029859 shutdown_connections 2024-02-19T22:17:53.791 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.788+0000 7fab97fb1700 1 --2- 172.21.15.184:0/1493029859 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fab7c02ff10 0x7fab7c0323d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:53.791 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.788+0000 7fab97fb1700 1 --2- 172.21.15.184:0/1493029859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab90060280 0x7fab90106780 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:53.791 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.788+0000 7fab97fb1700 1 -- 172.21.15.184:0/1493029859 >> 172.21.15.184:0/1493029859 conn(0x7fab900f0b70 msgr2=0x7fab900f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:53.791 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.788+0000 7fab97fb1700 1 -- 172.21.15.184:0/1493029859 shutdown_connections 2024-02-19T22:17:53.791 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:53.788+0000 7fab97fb1700 1 -- 172.21.15.184:0/1493029859 wait complete. 2024-02-19T22:17:53.791 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:54.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:51.917065+0000 mgr.smithi176.faoogw (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:54.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:53.787950+0000 mon.smithi176 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.184:0/1493029859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:55.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:55.473 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:56.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:55 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:53.917351+0000 mgr.smithi176.faoogw (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:57.385 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.381+0000 7f78e76bc700 1 -- 172.21.15.184:0/1407729469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78e00f5410 msgr2=0x7f78e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:57.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.381+0000 7f78e76bc700 1 --2- 172.21.15.184:0/1407729469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78e00f5410 0x7f78e00f5830 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f78d0009fb0 tx=0x7f78d0009b30).stop 2024-02-19T22:17:57.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.381+0000 7f78e76bc700 1 -- 172.21.15.184:0/1407729469 shutdown_connections 2024-02-19T22:17:57.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.381+0000 7f78e76bc700 1 --2- 172.21.15.184:0/1407729469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78e00f5410 0x7f78e00f5830 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:57.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.381+0000 7f78e76bc700 1 -- 172.21.15.184:0/1407729469 >> 172.21.15.184:0/1407729469 conn(0x7f78e00f0b70 msgr2=0x7f78e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:57.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.382+0000 7f78e76bc700 1 -- 172.21.15.184:0/1407729469 shutdown_connections 2024-02-19T22:17:57.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.382+0000 7f78e76bc700 1 -- 172.21.15.184:0/1407729469 wait complete. 2024-02-19T22:17:57.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.382+0000 7f78e76bc700 1 Processor -- start 2024-02-19T22:17:57.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.382+0000 7f78e76bc700 1 -- start start 2024-02-19T22:17:57.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.382+0000 7f78e76bc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78e00f5410 0x7f78e0109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:57.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.382+0000 7f78e76bc700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e0060240 con 0x7f78e00f5410 2024-02-19T22:17:57.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.383+0000 7f78e5458700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78e00f5410 0x7f78e0109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:57.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.383+0000 7f78e5458700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78e00f5410 0x7f78e0109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:46964/0 (socket says 172.21.15.184:46964) 2024-02-19T22:17:57.387 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.383+0000 7f78e5458700 1 -- 172.21.15.184:0/401021403 learned_addr learned my addr 172.21.15.184:0/401021403 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:17:57.387 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.383+0000 7f78e5458700 1 -- 172.21.15.184:0/401021403 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d0005040 con 0x7f78e00f5410 2024-02-19T22:17:57.387 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.383+0000 7f78e5458700 1 --2- 172.21.15.184:0/401021403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78e00f5410 0x7f78e0109510 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f78d0005010 tx=0x7f78d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:57.387 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.383+0000 7f78d5ffb700 1 -- 172.21.15.184:0/401021403 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78d0009320 con 0x7f78e00f5410 2024-02-19T22:17:57.387 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.383+0000 7f78e76bc700 1 -- 172.21.15.184:0/401021403 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e0060440 con 0x7f78e00f5410 2024-02-19T22:17:57.387 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.383+0000 7f78d5ffb700 1 -- 172.21.15.184:0/401021403 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78d0013380 con 0x7f78e00f5410 2024-02-19T22:17:57.387 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.383+0000 7f78d5ffb700 1 -- 172.21.15.184:0/401021403 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78d0014e50 con 0x7f78e00f5410 2024-02-19T22:17:57.387 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.383+0000 7f78e76bc700 1 -- 172.21.15.184:0/401021403 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e0109bc0 con 0x7f78e00f5410 2024-02-19T22:17:57.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.384+0000 7f78d5ffb700 1 -- 172.21.15.184:0/401021403 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78d00263a0 con 0x7f78e00f5410 2024-02-19T22:17:57.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.384+0000 7f78d5ffb700 1 --2- 172.21.15.184:0/401021403 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f78cc02ff10 0x7f78cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:57.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.384+0000 7f78e76bc700 1 -- 172.21.15.184:0/401021403 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e0045ad0 con 0x7f78e00f5410 2024-02-19T22:17:57.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.384+0000 7f78d5ffb700 1 -- 172.21.15.184:0/401021403 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78d0040fd0 con 0x7f78e00f5410 2024-02-19T22:17:57.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.385+0000 7f78e4c57700 1 --2- 172.21.15.184:0/401021403 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f78cc02ff10 0x7f78cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:57.388 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.385+0000 7f78e4c57700 1 --2- 172.21.15.184:0/401021403 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f78cc02ff10 0x7f78cc0323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f78dc0096a0 tx=0x7f78dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:57.391 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.389+0000 7f78d5ffb700 1 -- 172.21.15.184:0/401021403 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78d0012920 con 0x7f78e00f5410 2024-02-19T22:17:57.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.594+0000 7f78e76bc700 1 -- 172.21.15.184:0/401021403 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78e010a3e0 con 0x7f78e00f5410 2024-02-19T22:17:57.596 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.594+0000 7f78d5ffb700 1 -- 172.21.15.184:0/401021403 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f78d002b020 con 0x7f78e00f5410 2024-02-19T22:17:57.596 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:17:57.597 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:57.598 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.596+0000 7f78e76bc700 1 -- 172.21.15.184:0/401021403 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f78cc02ff10 msgr2=0x7f78cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:57.598 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.596+0000 7f78e76bc700 1 --2- 172.21.15.184:0/401021403 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f78cc02ff10 0x7f78cc0323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f78dc0096a0 tx=0x7f78dc006b40).stop 2024-02-19T22:17:57.598 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.596+0000 7f78e76bc700 1 -- 172.21.15.184:0/401021403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78e00f5410 msgr2=0x7f78e0109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:57.599 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.596+0000 7f78e76bc700 1 --2- 172.21.15.184:0/401021403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78e00f5410 0x7f78e0109510 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f78d0005010 tx=0x7f78d000b040).stop 2024-02-19T22:17:57.599 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.596+0000 7f78e76bc700 1 -- 172.21.15.184:0/401021403 shutdown_connections 2024-02-19T22:17:57.599 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.596+0000 7f78e76bc700 1 --2- 172.21.15.184:0/401021403 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f78cc02ff10 0x7f78cc0323d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:57.599 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.596+0000 7f78e76bc700 1 --2- 172.21.15.184:0/401021403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f78e00f5410 0x7f78e0109510 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:57.599 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.596+0000 7f78e76bc700 1 -- 172.21.15.184:0/401021403 >> 172.21.15.184:0/401021403 conn(0x7f78e00f0b70 msgr2=0x7f78e00f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:57.599 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.596+0000 7f78e76bc700 1 -- 172.21.15.184:0/401021403 shutdown_connections 2024-02-19T22:17:57.599 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:17:57.596+0000 7f78e76bc700 1 -- 172.21.15.184:0/401021403 wait complete. 2024-02-19T22:17:57.599 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:17:58.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:55.917711+0000 mgr.smithi176.faoogw (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:58.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:17:57.595828+0000 mon.smithi176 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.184:0/401021403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:59.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:17:58 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:57.918082+0000 mgr.smithi176.faoogw (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:59.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:59.361 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:01.243 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.241+0000 7f8851971700 1 -- 172.21.15.184:0/3092662605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f884c060220 msgr2=0x7f884c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:01.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.241+0000 7f8851971700 1 --2- 172.21.15.184:0/3092662605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f884c060220 0x7f884c0f9150 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f883c009fb0 tx=0x7f883c009b30).stop 2024-02-19T22:18:01.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.241+0000 7f8851971700 1 -- 172.21.15.184:0/3092662605 shutdown_connections 2024-02-19T22:18:01.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.241+0000 7f8851971700 1 --2- 172.21.15.184:0/3092662605 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f884c060220 0x7f884c0f9150 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:01.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.241+0000 7f8851971700 1 -- 172.21.15.184:0/3092662605 >> 172.21.15.184:0/3092662605 conn(0x7f884c0f0b70 msgr2=0x7f884c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:01.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.241+0000 7f8851971700 1 -- 172.21.15.184:0/3092662605 shutdown_connections 2024-02-19T22:18:01.244 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.241+0000 7f8851971700 1 -- 172.21.15.184:0/3092662605 wait complete. 2024-02-19T22:18:01.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.242+0000 7f8851971700 1 Processor -- start 2024-02-19T22:18:01.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.242+0000 7f8851971700 1 -- start start 2024-02-19T22:18:01.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.242+0000 7f8851971700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f884c060220 0x7f884c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:01.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.242+0000 7f8851971700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884c106cb0 con 0x7f884c060220 2024-02-19T22:18:01.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.242+0000 7f884affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f884c060220 0x7f884c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:01.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.242+0000 7f884affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f884c060220 0x7f884c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:46984/0 (socket says 172.21.15.184:46984) 2024-02-19T22:18:01.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.242+0000 7f884affd700 1 -- 172.21.15.184:0/1626667208 learned_addr learned my addr 172.21.15.184:0/1626667208 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:01.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.243+0000 7f884affd700 1 -- 172.21.15.184:0/1626667208 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883c005040 con 0x7f884c060220 2024-02-19T22:18:01.246 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.243+0000 7f884affd700 1 --2- 172.21.15.184:0/1626667208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f884c060220 0x7f884c1067c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f883c0089f0 tx=0x7f883c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:01.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.243+0000 7f88437fe700 1 -- 172.21.15.184:0/1626667208 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f883c009320 con 0x7f884c060220 2024-02-19T22:18:01.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.243+0000 7f8851971700 1 -- 172.21.15.184:0/1626667208 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f884c106eb0 con 0x7f884c060220 2024-02-19T22:18:01.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.243+0000 7f88437fe700 1 -- 172.21.15.184:0/1626667208 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f883c013380 con 0x7f884c060220 2024-02-19T22:18:01.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.243+0000 7f88437fe700 1 -- 172.21.15.184:0/1626667208 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f883c014e20 con 0x7f884c060220 2024-02-19T22:18:01.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.243+0000 7f8851971700 1 -- 172.21.15.184:0/1626667208 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f884c107340 con 0x7f884c060220 2024-02-19T22:18:01.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.244+0000 7f88437fe700 1 -- 172.21.15.184:0/1626667208 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f883c014420 con 0x7f884c060220 2024-02-19T22:18:01.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.244+0000 7f8851971700 1 -- 172.21.15.184:0/1626667208 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f884c045ad0 con 0x7f884c060220 2024-02-19T22:18:01.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.244+0000 7f88437fe700 1 --2- 172.21.15.184:0/1626667208 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f883802ff60 0x7f8838032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:01.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.244+0000 7f88437fe700 1 -- 172.21.15.184:0/1626667208 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f883c040e20 con 0x7f884c060220 2024-02-19T22:18:01.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.244+0000 7f884a7fc700 1 --2- 172.21.15.184:0/1626667208 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f883802ff60 0x7f8838032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:01.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.245+0000 7f884a7fc700 1 --2- 172.21.15.184:0/1626667208 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f883802ff60 0x7f8838032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f88340096a0 tx=0x7f8834006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:01.251 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.249+0000 7f88437fe700 1 -- 172.21.15.184:0/1626667208 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f883c01d980 con 0x7f884c060220 2024-02-19T22:18:01.457 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.455+0000 7f8851971700 1 -- 172.21.15.184:0/1626667208 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f884c1081f0 con 0x7f884c060220 2024-02-19T22:18:01.458 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.456+0000 7f88437fe700 1 -- 172.21.15.184:0/1626667208 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f883c010230 con 0x7f884c060220 2024-02-19T22:18:01.458 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:01.458 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:01.460 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.458+0000 7f8851971700 1 -- 172.21.15.184:0/1626667208 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f883802ff60 msgr2=0x7f8838032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:01.460 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.458+0000 7f8851971700 1 --2- 172.21.15.184:0/1626667208 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f883802ff60 0x7f8838032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f88340096a0 tx=0x7f8834006b40).stop 2024-02-19T22:18:01.460 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.458+0000 7f8851971700 1 -- 172.21.15.184:0/1626667208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f884c060220 msgr2=0x7f884c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:01.460 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.458+0000 7f8851971700 1 --2- 172.21.15.184:0/1626667208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f884c060220 0x7f884c1067c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f883c0089f0 tx=0x7f883c00b040).stop 2024-02-19T22:18:01.460 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.458+0000 7f8851971700 1 -- 172.21.15.184:0/1626667208 shutdown_connections 2024-02-19T22:18:01.460 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.458+0000 7f8851971700 1 --2- 172.21.15.184:0/1626667208 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f883802ff60 0x7f8838032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:01.460 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.458+0000 7f8851971700 1 --2- 172.21.15.184:0/1626667208 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f884c060220 0x7f884c1067c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:01.460 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.458+0000 7f8851971700 1 -- 172.21.15.184:0/1626667208 >> 172.21.15.184:0/1626667208 conn(0x7f884c0f0b70 msgr2=0x7f884c0f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:01.461 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.458+0000 7f8851971700 1 -- 172.21.15.184:0/1626667208 shutdown_connections 2024-02-19T22:18:01.461 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:01.458+0000 7f8851971700 1 -- 172.21.15.184:0/1626667208 wait complete. 2024-02-19T22:18:01.461 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:02.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:17:59.918398+0000 mgr.smithi176.faoogw (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:02.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:01.457378+0000 mon.smithi176 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.184:0/1626667208' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:03.100 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:03.101 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:04.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:01.918776+0000 mgr.smithi176.faoogw (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:04.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.925+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/567914684 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5f80f6510 msgr2=0x7fd5f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:04.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.925+0000 7fd5fe2d9700 1 --2- 172.21.15.184:0/567914684 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5f80f6510 0x7fd5f80f6930 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fd5ec009fb0 tx=0x7fd5ec009b30).stop 2024-02-19T22:18:04.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.925+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/567914684 shutdown_connections 2024-02-19T22:18:04.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.925+0000 7fd5fe2d9700 1 --2- 172.21.15.184:0/567914684 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5f80f6510 0x7fd5f80f6930 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:04.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.925+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/567914684 >> 172.21.15.184:0/567914684 conn(0x7fd5f80f1c70 msgr2=0x7fd5f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:04.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.925+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/567914684 shutdown_connections 2024-02-19T22:18:04.929 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.925+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/567914684 wait complete. 2024-02-19T22:18:04.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.925+0000 7fd5fe2d9700 1 Processor -- start 2024-02-19T22:18:04.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.926+0000 7fd5fe2d9700 1 -- start start 2024-02-19T22:18:04.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.926+0000 7fd5fe2d9700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5f80f6510 0x7fd5f806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:04.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.926+0000 7fd5fe2d9700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f8069e20 con 0x7fd5f80f6510 2024-02-19T22:18:04.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.926+0000 7fd5f7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5f80f6510 0x7fd5f806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:04.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.926+0000 7fd5f7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5f80f6510 0x7fd5f806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:47008/0 (socket says 172.21.15.184:47008) 2024-02-19T22:18:04.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.926+0000 7fd5f7fff700 1 -- 172.21.15.184:0/487601758 learned_addr learned my addr 172.21.15.184:0/487601758 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:04.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.926+0000 7fd5f7fff700 1 -- 172.21.15.184:0/487601758 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec005040 con 0x7fd5f80f6510 2024-02-19T22:18:04.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.926+0000 7fd5f7fff700 1 --2- 172.21.15.184:0/487601758 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5f80f6510 0x7fd5f806b9a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fd5ec005010 tx=0x7fd5ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:04.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.926+0000 7fd5f4ff9700 1 -- 172.21.15.184:0/487601758 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5ec009320 con 0x7fd5f80f6510 2024-02-19T22:18:04.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.927+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/487601758 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f8069fc0 con 0x7fd5f80f6510 2024-02-19T22:18:04.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.927+0000 7fd5f4ff9700 1 -- 172.21.15.184:0/487601758 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5ec013380 con 0x7fd5f80f6510 2024-02-19T22:18:04.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.927+0000 7fd5f4ff9700 1 -- 172.21.15.184:0/487601758 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5ec014e50 con 0x7fd5f80f6510 2024-02-19T22:18:04.930 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.927+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/487601758 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f806a450 con 0x7fd5f80f6510 2024-02-19T22:18:04.931 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.927+0000 7fd5f4ff9700 1 -- 172.21.15.184:0/487601758 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5ec0095f0 con 0x7fd5f80f6510 2024-02-19T22:18:04.931 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.927+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/487601758 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f8059100 con 0x7fd5f80f6510 2024-02-19T22:18:04.931 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.928+0000 7fd5f4ff9700 1 --2- 172.21.15.184:0/487601758 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd5e002ff10 0x7fd5e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:04.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.928+0000 7fd5f4ff9700 1 -- 172.21.15.184:0/487601758 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd5ec040ed0 con 0x7fd5f80f6510 2024-02-19T22:18:04.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.928+0000 7fd5f77fe700 1 --2- 172.21.15.184:0/487601758 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd5e002ff10 0x7fd5e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:04.932 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.928+0000 7fd5f77fe700 1 --2- 172.21.15.184:0/487601758 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd5e002ff10 0x7fd5e00323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fd5e80096a0 tx=0x7fd5e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:04.934 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:04.932+0000 7fd5f4ff9700 1 -- 172.21.15.184:0/487601758 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5ec014420 con 0x7fd5f80f6510 2024-02-19T22:18:05.140 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:05.138+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/487601758 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5f80fc0a0 con 0x7fd5f80f6510 2024-02-19T22:18:05.140 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:05.138+0000 7fd5f4ff9700 1 -- 172.21.15.184:0/487601758 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd5ec014420 con 0x7fd5f80f6510 2024-02-19T22:18:05.141 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:05.141 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:05.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:05.140+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/487601758 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd5e002ff10 msgr2=0x7fd5e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:05.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:05.140+0000 7fd5fe2d9700 1 --2- 172.21.15.184:0/487601758 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd5e002ff10 0x7fd5e00323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fd5e80096a0 tx=0x7fd5e8006b40).stop 2024-02-19T22:18:05.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:05.140+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/487601758 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5f80f6510 msgr2=0x7fd5f806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:05.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:05.140+0000 7fd5fe2d9700 1 --2- 172.21.15.184:0/487601758 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5f80f6510 0x7fd5f806b9a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fd5ec005010 tx=0x7fd5ec00b040).stop 2024-02-19T22:18:05.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:05.140+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/487601758 shutdown_connections 2024-02-19T22:18:05.142 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:05.140+0000 7fd5fe2d9700 1 --2- 172.21.15.184:0/487601758 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fd5e002ff10 0x7fd5e00323d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:05.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:05.140+0000 7fd5fe2d9700 1 --2- 172.21.15.184:0/487601758 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd5f80f6510 0x7fd5f806b9a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:05.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:05.140+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/487601758 >> 172.21.15.184:0/487601758 conn(0x7fd5f80f1c70 msgr2=0x7fd5f80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:05.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:05.140+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/487601758 shutdown_connections 2024-02-19T22:18:05.143 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:05.140+0000 7fd5fe2d9700 1 -- 172.21.15.184:0/487601758 wait complete. 2024-02-19T22:18:05.143 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:06.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:03.919129+0000 mgr.smithi176.faoogw (mgr.14162) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:06.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:05.139946+0000 mon.smithi176 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.184:0/487601758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:06.805 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:06.805 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:08.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:05.919458+0000 mgr.smithi176.faoogw (mgr.14162) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:08.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.683+0000 7f943c14a700 1 -- 172.21.15.184:0/2910910795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9434060280 msgr2=0x7f94340f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:08.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.683+0000 7f943c14a700 1 --2- 172.21.15.184:0/2910910795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9434060280 0x7f94340f90f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f9430009fb0 tx=0x7f9430009b30).stop 2024-02-19T22:18:08.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.683+0000 7f943c14a700 1 -- 172.21.15.184:0/2910910795 shutdown_connections 2024-02-19T22:18:08.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.683+0000 7f943c14a700 1 --2- 172.21.15.184:0/2910910795 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9434060280 0x7f94340f90f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:08.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.683+0000 7f943c14a700 1 -- 172.21.15.184:0/2910910795 >> 172.21.15.184:0/2910910795 conn(0x7f94340f0b70 msgr2=0x7f94340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:08.687 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.684+0000 7f943c14a700 1 -- 172.21.15.184:0/2910910795 shutdown_connections 2024-02-19T22:18:08.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.684+0000 7f943c14a700 1 -- 172.21.15.184:0/2910910795 wait complete. 2024-02-19T22:18:08.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.684+0000 7f943c14a700 1 Processor -- start 2024-02-19T22:18:08.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.684+0000 7f943c14a700 1 -- start start 2024-02-19T22:18:08.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.684+0000 7f943c14a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9434060280 0x7f94340f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:08.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.684+0000 7f943c14a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94340f4ff0 con 0x7f9434060280 2024-02-19T22:18:08.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.685+0000 7f9439ee6700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9434060280 0x7f94340f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:08.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.685+0000 7f9439ee6700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9434060280 0x7f94340f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:35434/0 (socket says 172.21.15.184:35434) 2024-02-19T22:18:08.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.685+0000 7f9439ee6700 1 -- 172.21.15.184:0/3395004348 learned_addr learned my addr 172.21.15.184:0/3395004348 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:08.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.685+0000 7f9439ee6700 1 -- 172.21.15.184:0/3395004348 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9430005040 con 0x7f9434060280 2024-02-19T22:18:08.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.685+0000 7f9439ee6700 1 --2- 172.21.15.184:0/3395004348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9434060280 0x7f94340f6b70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f94300089f0 tx=0x7f943000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:08.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.685+0000 7f942a7fc700 1 -- 172.21.15.184:0/3395004348 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9430009320 con 0x7f9434060280 2024-02-19T22:18:08.688 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.685+0000 7f943c14a700 1 -- 172.21.15.184:0/3395004348 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94340f5190 con 0x7f9434060280 2024-02-19T22:18:08.689 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.685+0000 7f942a7fc700 1 -- 172.21.15.184:0/3395004348 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9430013380 con 0x7f9434060280 2024-02-19T22:18:08.689 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.685+0000 7f942a7fc700 1 -- 172.21.15.184:0/3395004348 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9430014e20 con 0x7f9434060280 2024-02-19T22:18:08.689 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.685+0000 7f943c14a700 1 -- 172.21.15.184:0/3395004348 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94340f5620 con 0x7f9434060280 2024-02-19T22:18:08.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.686+0000 7f942a7fc700 1 -- 172.21.15.184:0/3395004348 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94300263a0 con 0x7f9434060280 2024-02-19T22:18:08.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.686+0000 7f942a7fc700 1 --2- 172.21.15.184:0/3395004348 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f942002ff10 0x7f94200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:08.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.686+0000 7f942a7fc700 1 -- 172.21.15.184:0/3395004348 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9430040f20 con 0x7f9434060280 2024-02-19T22:18:08.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.686+0000 7f943c14a700 1 -- 172.21.15.184:0/3395004348 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9434059100 con 0x7f9434060280 2024-02-19T22:18:08.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.686+0000 7f94396e5700 1 --2- 172.21.15.184:0/3395004348 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f942002ff10 0x7f94200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:08.690 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.687+0000 7f94396e5700 1 --2- 172.21.15.184:0/3395004348 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f942002ff10 0x7f94200323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f94240096a0 tx=0x7f9424006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:08.693 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.691+0000 7f942a7fc700 1 -- 172.21.15.184:0/3395004348 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9430012920 con 0x7f9434060280 2024-02-19T22:18:08.897 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.895+0000 7f943c14a700 1 -- 172.21.15.184:0/3395004348 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9434045ad0 con 0x7f9434060280 2024-02-19T22:18:08.898 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.896+0000 7f942a7fc700 1 -- 172.21.15.184:0/3395004348 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f943002b020 con 0x7f9434060280 2024-02-19T22:18:08.898 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:08.898 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:08.900 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.897+0000 7f943c14a700 1 -- 172.21.15.184:0/3395004348 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f942002ff10 msgr2=0x7f94200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:08.900 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.897+0000 7f943c14a700 1 --2- 172.21.15.184:0/3395004348 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f942002ff10 0x7f94200323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f94240096a0 tx=0x7f9424006b40).stop 2024-02-19T22:18:08.900 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.897+0000 7f943c14a700 1 -- 172.21.15.184:0/3395004348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9434060280 msgr2=0x7f94340f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:08.900 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.897+0000 7f943c14a700 1 --2- 172.21.15.184:0/3395004348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9434060280 0x7f94340f6b70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f94300089f0 tx=0x7f943000b040).stop 2024-02-19T22:18:08.900 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.897+0000 7f943c14a700 1 -- 172.21.15.184:0/3395004348 shutdown_connections 2024-02-19T22:18:08.900 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.898+0000 7f943c14a700 1 --2- 172.21.15.184:0/3395004348 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f942002ff10 0x7f94200323d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:08.900 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.898+0000 7f943c14a700 1 --2- 172.21.15.184:0/3395004348 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f9434060280 0x7f94340f6b70 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:08.900 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.898+0000 7f943c14a700 1 -- 172.21.15.184:0/3395004348 >> 172.21.15.184:0/3395004348 conn(0x7f94340f0b70 msgr2=0x7f94340f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:08.900 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.898+0000 7f943c14a700 1 -- 172.21.15.184:0/3395004348 shutdown_connections 2024-02-19T22:18:08.901 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:08.898+0000 7f943c14a700 1 -- 172.21.15.184:0/3395004348 wait complete. 2024-02-19T22:18:08.901 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:10.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:07.919777+0000 mgr.smithi176.faoogw (mgr.14162) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:10.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:08.897345+0000 mon.smithi176 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.184:0/3395004348' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:10.506 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:10.507 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:12.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:09.920041+0000 mgr.smithi176.faoogw (mgr.14162) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:12.430 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.426+0000 7f33772d3700 1 -- 172.21.15.184:0/1546361553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33700f5410 msgr2=0x7f33700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:12.430 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.426+0000 7f33772d3700 1 --2- 172.21.15.184:0/1546361553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33700f5410 0x7f33700f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f3360009fb0 tx=0x7f3360009b30).stop 2024-02-19T22:18:12.430 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.426+0000 7f33772d3700 1 -- 172.21.15.184:0/1546361553 shutdown_connections 2024-02-19T22:18:12.430 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.426+0000 7f33772d3700 1 --2- 172.21.15.184:0/1546361553 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33700f5410 0x7f33700f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:12.430 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.426+0000 7f33772d3700 1 -- 172.21.15.184:0/1546361553 >> 172.21.15.184:0/1546361553 conn(0x7f33700f0b70 msgr2=0x7f33700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:12.430 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.426+0000 7f33772d3700 1 -- 172.21.15.184:0/1546361553 shutdown_connections 2024-02-19T22:18:12.430 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.426+0000 7f33772d3700 1 -- 172.21.15.184:0/1546361553 wait complete. 2024-02-19T22:18:12.430 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.426+0000 7f33772d3700 1 Processor -- start 2024-02-19T22:18:12.430 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.426+0000 7f33772d3700 1 -- start start 2024-02-19T22:18:12.431 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.426+0000 7f33772d3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33700f5410 0x7f33701067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:12.431 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.426+0000 7f33772d3700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3370106cc0 con 0x7f33700f5410 2024-02-19T22:18:12.431 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.427+0000 7f337506f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33700f5410 0x7f33701067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:12.431 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.427+0000 7f337506f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33700f5410 0x7f33701067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:35458/0 (socket says 172.21.15.184:35458) 2024-02-19T22:18:12.431 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.427+0000 7f337506f700 1 -- 172.21.15.184:0/3568490248 learned_addr learned my addr 172.21.15.184:0/3568490248 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:12.431 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.427+0000 7f337506f700 1 -- 172.21.15.184:0/3568490248 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3360005040 con 0x7f33700f5410 2024-02-19T22:18:12.431 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.427+0000 7f337506f700 1 --2- 172.21.15.184:0/3568490248 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33700f5410 0x7f33701067d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f3360005010 tx=0x7f336000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:12.431 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.427+0000 7f3365ffb700 1 -- 172.21.15.184:0/3568490248 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3360009320 con 0x7f33700f5410 2024-02-19T22:18:12.431 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.427+0000 7f33772d3700 1 -- 172.21.15.184:0/3568490248 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3370106ec0 con 0x7f33700f5410 2024-02-19T22:18:12.431 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.427+0000 7f3365ffb700 1 -- 172.21.15.184:0/3568490248 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3360013380 con 0x7f33700f5410 2024-02-19T22:18:12.431 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.427+0000 7f3365ffb700 1 -- 172.21.15.184:0/3568490248 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3360014e50 con 0x7f33700f5410 2024-02-19T22:18:12.431 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.427+0000 7f33772d3700 1 -- 172.21.15.184:0/3568490248 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3370107350 con 0x7f33700f5410 2024-02-19T22:18:12.432 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.428+0000 7f3365ffb700 1 -- 172.21.15.184:0/3568490248 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3360014420 con 0x7f33700f5410 2024-02-19T22:18:12.432 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.428+0000 7f33772d3700 1 -- 172.21.15.184:0/3568490248 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3370045ad0 con 0x7f33700f5410 2024-02-19T22:18:12.432 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.428+0000 7f3365ffb700 1 --2- 172.21.15.184:0/3568490248 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f335c02ff10 0x7f335c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:12.433 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.428+0000 7f3365ffb700 1 -- 172.21.15.184:0/3568490248 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3360040f70 con 0x7f33700f5410 2024-02-19T22:18:12.433 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.429+0000 7f337486e700 1 --2- 172.21.15.184:0/3568490248 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f335c02ff10 0x7f335c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:12.433 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.429+0000 7f337486e700 1 --2- 172.21.15.184:0/3568490248 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f335c02ff10 0x7f335c0323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f336c0096a0 tx=0x7f336c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:12.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.433+0000 7f3365ffb700 1 -- 172.21.15.184:0/3568490248 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33600095f0 con 0x7f33700f5410 2024-02-19T22:18:12.638 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.636+0000 7f33772d3700 1 -- 172.21.15.184:0/3568490248 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3370108200 con 0x7f33700f5410 2024-02-19T22:18:12.639 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.637+0000 7f3365ffb700 1 -- 172.21.15.184:0/3568490248 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f336001db70 con 0x7f33700f5410 2024-02-19T22:18:12.639 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:12.639 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:12.641 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.639+0000 7f33772d3700 1 -- 172.21.15.184:0/3568490248 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f335c02ff10 msgr2=0x7f335c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:12.641 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.639+0000 7f33772d3700 1 --2- 172.21.15.184:0/3568490248 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f335c02ff10 0x7f335c0323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f336c0096a0 tx=0x7f336c006b40).stop 2024-02-19T22:18:12.641 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.639+0000 7f33772d3700 1 -- 172.21.15.184:0/3568490248 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33700f5410 msgr2=0x7f33701067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:12.641 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.639+0000 7f33772d3700 1 --2- 172.21.15.184:0/3568490248 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33700f5410 0x7f33701067d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f3360005010 tx=0x7f336000b040).stop 2024-02-19T22:18:12.641 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.639+0000 7f33772d3700 1 -- 172.21.15.184:0/3568490248 shutdown_connections 2024-02-19T22:18:12.641 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.639+0000 7f33772d3700 1 --2- 172.21.15.184:0/3568490248 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f335c02ff10 0x7f335c0323d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:12.641 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.639+0000 7f33772d3700 1 --2- 172.21.15.184:0/3568490248 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33700f5410 0x7f33701067d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:12.641 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.639+0000 7f33772d3700 1 -- 172.21.15.184:0/3568490248 >> 172.21.15.184:0/3568490248 conn(0x7f33700f0b70 msgr2=0x7f337005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:12.641 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.639+0000 7f33772d3700 1 -- 172.21.15.184:0/3568490248 shutdown_connections 2024-02-19T22:18:12.642 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:12.639+0000 7f33772d3700 1 -- 172.21.15.184:0/3568490248 wait complete. 2024-02-19T22:18:12.642 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:13.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:12 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:12.638587+0000 mon.smithi176 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.184:0/3568490248' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:14.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:13 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:11.920403+0000 mgr.smithi176.faoogw (mgr.14162) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:14.411 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:14.411 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:16.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:13.920686+0000 mgr.smithi176.faoogw (mgr.14162) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:16.359 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.356+0000 7f66755a2700 1 -- 172.21.15.184:0/1013547104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66700f4b00 msgr2=0x7f66700f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:16.359 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.356+0000 7f66755a2700 1 --2- 172.21.15.184:0/1013547104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66700f4b00 0x7f66700f4f20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f6668009fb0 tx=0x7f6668009b30).stop 2024-02-19T22:18:16.360 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.356+0000 7f66755a2700 1 -- 172.21.15.184:0/1013547104 shutdown_connections 2024-02-19T22:18:16.360 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.356+0000 7f66755a2700 1 --2- 172.21.15.184:0/1013547104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66700f4b00 0x7f66700f4f20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:16.360 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.356+0000 7f66755a2700 1 -- 172.21.15.184:0/1013547104 >> 172.21.15.184:0/1013547104 conn(0x7f66700f0260 msgr2=0x7f66700f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:16.360 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.356+0000 7f66755a2700 1 -- 172.21.15.184:0/1013547104 shutdown_connections 2024-02-19T22:18:16.360 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.356+0000 7f66755a2700 1 -- 172.21.15.184:0/1013547104 wait complete. 2024-02-19T22:18:16.360 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.357+0000 7f66755a2700 1 Processor -- start 2024-02-19T22:18:16.360 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.357+0000 7f66755a2700 1 -- start start 2024-02-19T22:18:16.360 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.357+0000 7f66755a2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66700f4b00 0x7f6670106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:16.360 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.357+0000 7f66755a2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6670106d10 con 0x7f66700f4b00 2024-02-19T22:18:16.360 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.357+0000 7f666effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66700f4b00 0x7f6670106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:16.360 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.357+0000 7f666effd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66700f4b00 0x7f6670106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:35474/0 (socket says 172.21.15.184:35474) 2024-02-19T22:18:16.360 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.357+0000 7f666effd700 1 -- 172.21.15.184:0/1072213560 learned_addr learned my addr 172.21.15.184:0/1072213560 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:16.362 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.358+0000 7f666effd700 1 -- 172.21.15.184:0/1072213560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6668005040 con 0x7f66700f4b00 2024-02-19T22:18:16.362 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.358+0000 7f666effd700 1 --2- 172.21.15.184:0/1072213560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66700f4b00 0x7f6670106820 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f6668005010 tx=0x7f666800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:16.362 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.358+0000 7f66677fe700 1 -- 172.21.15.184:0/1072213560 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6668009320 con 0x7f66700f4b00 2024-02-19T22:18:16.362 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.358+0000 7f66677fe700 1 -- 172.21.15.184:0/1072213560 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6668013380 con 0x7f66700f4b00 2024-02-19T22:18:16.362 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.358+0000 7f66755a2700 1 -- 172.21.15.184:0/1072213560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6670106f10 con 0x7f66700f4b00 2024-02-19T22:18:16.362 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.358+0000 7f66677fe700 1 -- 172.21.15.184:0/1072213560 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6668014e50 con 0x7f66700f4b00 2024-02-19T22:18:16.362 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.358+0000 7f66755a2700 1 -- 172.21.15.184:0/1072213560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66701073a0 con 0x7f66700f4b00 2024-02-19T22:18:16.362 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.359+0000 7f66677fe700 1 -- 172.21.15.184:0/1072213560 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66680095f0 con 0x7f66700f4b00 2024-02-19T22:18:16.363 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.359+0000 7f66755a2700 1 -- 172.21.15.184:0/1072213560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6670045ad0 con 0x7f66700f4b00 2024-02-19T22:18:16.363 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.359+0000 7f66677fe700 1 --2- 172.21.15.184:0/1072213560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f665c02ff10 0x7f665c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:16.363 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.359+0000 7f66677fe700 1 -- 172.21.15.184:0/1072213560 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6668040f80 con 0x7f66700f4b00 2024-02-19T22:18:16.363 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.360+0000 7f666e7fc700 1 --2- 172.21.15.184:0/1072213560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f665c02ff10 0x7f665c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:16.363 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.360+0000 7f666e7fc700 1 --2- 172.21.15.184:0/1072213560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f665c02ff10 0x7f665c0323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f66580096a0 tx=0x7f6658006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:16.366 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.364+0000 7f66677fe700 1 -- 172.21.15.184:0/1072213560 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6668014420 con 0x7f66700f4b00 2024-02-19T22:18:16.574 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.572+0000 7f66755a2700 1 -- 172.21.15.184:0/1072213560 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66700fa710 con 0x7f66700f4b00 2024-02-19T22:18:16.575 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.573+0000 7f66677fe700 1 -- 172.21.15.184:0/1072213560 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6668010230 con 0x7f66700f4b00 2024-02-19T22:18:16.576 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:16.576 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:16.577 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.575+0000 7f66755a2700 1 -- 172.21.15.184:0/1072213560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f665c02ff10 msgr2=0x7f665c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:16.577 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.575+0000 7f66755a2700 1 --2- 172.21.15.184:0/1072213560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f665c02ff10 0x7f665c0323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f66580096a0 tx=0x7f6658006b40).stop 2024-02-19T22:18:16.577 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.575+0000 7f66755a2700 1 -- 172.21.15.184:0/1072213560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66700f4b00 msgr2=0x7f6670106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:16.577 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.575+0000 7f66755a2700 1 --2- 172.21.15.184:0/1072213560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66700f4b00 0x7f6670106820 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f6668005010 tx=0x7f666800b040).stop 2024-02-19T22:18:16.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.575+0000 7f66755a2700 1 -- 172.21.15.184:0/1072213560 shutdown_connections 2024-02-19T22:18:16.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.575+0000 7f66755a2700 1 --2- 172.21.15.184:0/1072213560 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f665c02ff10 0x7f665c0323d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:16.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.575+0000 7f66755a2700 1 --2- 172.21.15.184:0/1072213560 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66700f4b00 0x7f6670106820 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:16.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.575+0000 7f66755a2700 1 -- 172.21.15.184:0/1072213560 >> 172.21.15.184:0/1072213560 conn(0x7f66700f0260 msgr2=0x7f66700f1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:16.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.575+0000 7f66755a2700 1 -- 172.21.15.184:0/1072213560 shutdown_connections 2024-02-19T22:18:16.578 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:16.575+0000 7f66755a2700 1 -- 172.21.15.184:0/1072213560 wait complete. 2024-02-19T22:18:16.578 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:17.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:16 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:16.574775+0000 mon.smithi176 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.184:0/1072213560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:18.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:17 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:15.921044+0000 mgr.smithi176.faoogw (mgr.14162) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:18.348 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:18.349 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:20.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:17.921411+0000 mgr.smithi176.faoogw (mgr.14162) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:20.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.251+0000 7fb152459700 1 -- 172.21.15.184:0/3118766379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb14c0f6510 msgr2=0x7fb14c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:20.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.251+0000 7fb152459700 1 --2- 172.21.15.184:0/3118766379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb14c0f6510 0x7fb14c0f6930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fb13c009fb0 tx=0x7fb13c009b30).stop 2024-02-19T22:18:20.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.251+0000 7fb152459700 1 -- 172.21.15.184:0/3118766379 shutdown_connections 2024-02-19T22:18:20.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.251+0000 7fb152459700 1 --2- 172.21.15.184:0/3118766379 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb14c0f6510 0x7fb14c0f6930 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:20.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.251+0000 7fb152459700 1 -- 172.21.15.184:0/3118766379 >> 172.21.15.184:0/3118766379 conn(0x7fb14c0f1c70 msgr2=0x7fb14c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:20.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.251+0000 7fb152459700 1 -- 172.21.15.184:0/3118766379 shutdown_connections 2024-02-19T22:18:20.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.251+0000 7fb152459700 1 -- 172.21.15.184:0/3118766379 wait complete. 2024-02-19T22:18:20.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.252+0000 7fb152459700 1 Processor -- start 2024-02-19T22:18:20.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.252+0000 7fb152459700 1 -- start start 2024-02-19T22:18:20.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.252+0000 7fb152459700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb14c0f6510 0x7fb14c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:20.255 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.252+0000 7fb152459700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14c106cd0 con 0x7fb14c0f6510 2024-02-19T22:18:20.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.252+0000 7fb14bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb14c0f6510 0x7fb14c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:20.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.252+0000 7fb14bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb14c0f6510 0x7fb14c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:53346/0 (socket says 172.21.15.184:53346) 2024-02-19T22:18:20.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.252+0000 7fb14bfff700 1 -- 172.21.15.184:0/2981815715 learned_addr learned my addr 172.21.15.184:0/2981815715 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:20.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.253+0000 7fb14bfff700 1 -- 172.21.15.184:0/2981815715 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c005040 con 0x7fb14c0f6510 2024-02-19T22:18:20.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.253+0000 7fb14bfff700 1 --2- 172.21.15.184:0/2981815715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb14c0f6510 0x7fb14c1067e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fb13c005010 tx=0x7fb13c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:20.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.253+0000 7fb148ff9700 1 -- 172.21.15.184:0/2981815715 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb13c009320 con 0x7fb14c0f6510 2024-02-19T22:18:20.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.253+0000 7fb152459700 1 -- 172.21.15.184:0/2981815715 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14c106ed0 con 0x7fb14c0f6510 2024-02-19T22:18:20.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.253+0000 7fb148ff9700 1 -- 172.21.15.184:0/2981815715 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb13c013380 con 0x7fb14c0f6510 2024-02-19T22:18:20.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.253+0000 7fb148ff9700 1 -- 172.21.15.184:0/2981815715 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb13c014e50 con 0x7fb14c0f6510 2024-02-19T22:18:20.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.253+0000 7fb152459700 1 -- 172.21.15.184:0/2981815715 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14c107360 con 0x7fb14c0f6510 2024-02-19T22:18:20.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.254+0000 7fb148ff9700 1 -- 172.21.15.184:0/2981815715 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb13c014420 con 0x7fb14c0f6510 2024-02-19T22:18:20.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.254+0000 7fb152459700 1 -- 172.21.15.184:0/2981815715 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14c045ad0 con 0x7fb14c0f6510 2024-02-19T22:18:20.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.254+0000 7fb148ff9700 1 --2- 172.21.15.184:0/2981815715 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb13402ff10 0x7fb1340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:20.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.254+0000 7fb148ff9700 1 -- 172.21.15.184:0/2981815715 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb13c040f40 con 0x7fb14c0f6510 2024-02-19T22:18:20.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.254+0000 7fb14b7fe700 1 --2- 172.21.15.184:0/2981815715 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb13402ff10 0x7fb1340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:20.258 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.255+0000 7fb14b7fe700 1 --2- 172.21.15.184:0/2981815715 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb13402ff10 0x7fb1340323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb1400096a0 tx=0x7fb140006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:20.260 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.259+0000 7fb148ff9700 1 -- 172.21.15.184:0/2981815715 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb13c0095f0 con 0x7fb14c0f6510 2024-02-19T22:18:20.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.466+0000 7fb152459700 1 -- 172.21.15.184:0/2981815715 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb14c108210 con 0x7fb14c0f6510 2024-02-19T22:18:20.468 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.466+0000 7fb148ff9700 1 -- 172.21.15.184:0/2981815715 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb13c01db70 con 0x7fb14c0f6510 2024-02-19T22:18:20.469 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:20.469 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:20.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.468+0000 7fb152459700 1 -- 172.21.15.184:0/2981815715 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb13402ff10 msgr2=0x7fb1340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:20.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.468+0000 7fb152459700 1 --2- 172.21.15.184:0/2981815715 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb13402ff10 0x7fb1340323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb1400096a0 tx=0x7fb140006b40).stop 2024-02-19T22:18:20.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.468+0000 7fb152459700 1 -- 172.21.15.184:0/2981815715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb14c0f6510 msgr2=0x7fb14c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:20.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.468+0000 7fb152459700 1 --2- 172.21.15.184:0/2981815715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb14c0f6510 0x7fb14c1067e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fb13c005010 tx=0x7fb13c00b040).stop 2024-02-19T22:18:20.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.468+0000 7fb152459700 1 -- 172.21.15.184:0/2981815715 shutdown_connections 2024-02-19T22:18:20.470 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.468+0000 7fb152459700 1 --2- 172.21.15.184:0/2981815715 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb13402ff10 0x7fb1340323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:20.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.468+0000 7fb152459700 1 --2- 172.21.15.184:0/2981815715 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb14c0f6510 0x7fb14c1067e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:20.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.468+0000 7fb152459700 1 -- 172.21.15.184:0/2981815715 >> 172.21.15.184:0/2981815715 conn(0x7fb14c0f1c70 msgr2=0x7fb14c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:20.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.468+0000 7fb152459700 1 -- 172.21.15.184:0/2981815715 shutdown_connections 2024-02-19T22:18:20.471 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:20.468+0000 7fb152459700 1 -- 172.21.15.184:0/2981815715 wait complete. 2024-02-19T22:18:20.471 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:21.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:20 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:20.467710+0000 mon.smithi176 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.184:0/2981815715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:22.101 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:22.102 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:22.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:19.921763+0000 mgr.smithi176.faoogw (mgr.14162) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:23.946 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.943+0000 7fcfcbd32700 1 -- 172.21.15.184:0/2586669448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfc40f53f0 msgr2=0x7fcfc40f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:23.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.943+0000 7fcfcbd32700 1 --2- 172.21.15.184:0/2586669448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfc40f53f0 0x7fcfc40f5810 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fcfb8009fb0 tx=0x7fcfb8009b30).stop 2024-02-19T22:18:23.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.943+0000 7fcfcbd32700 1 -- 172.21.15.184:0/2586669448 shutdown_connections 2024-02-19T22:18:23.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.943+0000 7fcfcbd32700 1 --2- 172.21.15.184:0/2586669448 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfc40f53f0 0x7fcfc40f5810 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:23.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.943+0000 7fcfcbd32700 1 -- 172.21.15.184:0/2586669448 >> 172.21.15.184:0/2586669448 conn(0x7fcfc40f0b70 msgr2=0x7fcfc40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:23.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.943+0000 7fcfcbd32700 1 -- 172.21.15.184:0/2586669448 shutdown_connections 2024-02-19T22:18:23.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.943+0000 7fcfcbd32700 1 -- 172.21.15.184:0/2586669448 wait complete. 2024-02-19T22:18:23.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.944+0000 7fcfcbd32700 1 Processor -- start 2024-02-19T22:18:23.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.944+0000 7fcfcbd32700 1 -- start start 2024-02-19T22:18:23.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.944+0000 7fcfcbd32700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfc40f53f0 0x7fcfc41094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:23.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.944+0000 7fcfcbd32700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc4060220 con 0x7fcfc40f53f0 2024-02-19T22:18:23.947 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.944+0000 7fcfc9ace700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfc40f53f0 0x7fcfc41094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:23.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.944+0000 7fcfc9ace700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfc40f53f0 0x7fcfc41094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:53370/0 (socket says 172.21.15.184:53370) 2024-02-19T22:18:23.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.944+0000 7fcfc9ace700 1 -- 172.21.15.184:0/336893171 learned_addr learned my addr 172.21.15.184:0/336893171 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:23.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.944+0000 7fcfc9ace700 1 -- 172.21.15.184:0/336893171 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb8005040 con 0x7fcfc40f53f0 2024-02-19T22:18:23.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.944+0000 7fcfc9ace700 1 --2- 172.21.15.184:0/336893171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfc40f53f0 0x7fcfc41094e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fcfb8005010 tx=0x7fcfb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:23.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.945+0000 7fcfb67fc700 1 -- 172.21.15.184:0/336893171 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfb8009320 con 0x7fcfc40f53f0 2024-02-19T22:18:23.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.945+0000 7fcfcbd32700 1 -- 172.21.15.184:0/336893171 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc4060420 con 0x7fcfc40f53f0 2024-02-19T22:18:23.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.945+0000 7fcfb67fc700 1 -- 172.21.15.184:0/336893171 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfb8013380 con 0x7fcfc40f53f0 2024-02-19T22:18:23.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.945+0000 7fcfb67fc700 1 -- 172.21.15.184:0/336893171 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfb8014e50 con 0x7fcfc40f53f0 2024-02-19T22:18:23.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.945+0000 7fcfcbd32700 1 -- 172.21.15.184:0/336893171 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc4109b90 con 0x7fcfc40f53f0 2024-02-19T22:18:23.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.945+0000 7fcfb67fc700 1 -- 172.21.15.184:0/336893171 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfb80095f0 con 0x7fcfc40f53f0 2024-02-19T22:18:23.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.945+0000 7fcfcbd32700 1 -- 172.21.15.184:0/336893171 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc4059100 con 0x7fcfc40f53f0 2024-02-19T22:18:23.948 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.946+0000 7fcfb67fc700 1 --2- 172.21.15.184:0/336893171 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcfb002ff10 0x7fcfb00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:23.949 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.946+0000 7fcfb67fc700 1 -- 172.21.15.184:0/336893171 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfb8040e90 con 0x7fcfc40f53f0 2024-02-19T22:18:23.949 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.946+0000 7fcfc92cd700 1 --2- 172.21.15.184:0/336893171 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcfb002ff10 0x7fcfb00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:23.949 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.946+0000 7fcfc92cd700 1 --2- 172.21.15.184:0/336893171 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcfb002ff10 0x7fcfb00323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fcfc00096a0 tx=0x7fcfc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:23.951 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:23.950+0000 7fcfb67fc700 1 -- 172.21.15.184:0/336893171 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfb8014420 con 0x7fcfc40f53f0 2024-02-19T22:18:24.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:21.922062+0000 mgr.smithi176.faoogw (mgr.14162) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:24.154 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:24.154+0000 7fcfcbd32700 1 -- 172.21.15.184:0/336893171 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfc4045ad0 con 0x7fcfc40f53f0 2024-02-19T22:18:24.155 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:24.154+0000 7fcfb67fc700 1 -- 172.21.15.184:0/336893171 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcfb8010230 con 0x7fcfc40f53f0 2024-02-19T22:18:24.156 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:24.156 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:24.157 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:24.156+0000 7fcfcbd32700 1 -- 172.21.15.184:0/336893171 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcfb002ff10 msgr2=0x7fcfb00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:24.157 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:24.156+0000 7fcfcbd32700 1 --2- 172.21.15.184:0/336893171 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcfb002ff10 0x7fcfb00323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fcfc00096a0 tx=0x7fcfc0006b40).stop 2024-02-19T22:18:24.157 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:24.156+0000 7fcfcbd32700 1 -- 172.21.15.184:0/336893171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfc40f53f0 msgr2=0x7fcfc41094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:24.157 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:24.156+0000 7fcfcbd32700 1 --2- 172.21.15.184:0/336893171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfc40f53f0 0x7fcfc41094e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fcfb8005010 tx=0x7fcfb800b040).stop 2024-02-19T22:18:24.157 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:24.156+0000 7fcfcbd32700 1 -- 172.21.15.184:0/336893171 shutdown_connections 2024-02-19T22:18:24.157 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:24.156+0000 7fcfcbd32700 1 --2- 172.21.15.184:0/336893171 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fcfb002ff10 0x7fcfb00323d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:24.157 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:24.156+0000 7fcfcbd32700 1 --2- 172.21.15.184:0/336893171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcfc40f53f0 0x7fcfc41094e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:24.157 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:24.156+0000 7fcfcbd32700 1 -- 172.21.15.184:0/336893171 >> 172.21.15.184:0/336893171 conn(0x7fcfc40f0b70 msgr2=0x7fcfc40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:24.157 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:24.156+0000 7fcfcbd32700 1 -- 172.21.15.184:0/336893171 shutdown_connections 2024-02-19T22:18:24.158 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:24.156+0000 7fcfcbd32700 1 -- 172.21.15.184:0/336893171 wait complete. 2024-02-19T22:18:24.158 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:25.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:24 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:24.154835+0000 mon.smithi176 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.184:0/336893171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:25.923 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:25.923 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:26.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:23.922355+0000 mgr.smithi176.faoogw (mgr.14162) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:27.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.863+0000 7f3954da0700 1 -- 172.21.15.184:0/2764883270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39500f51a0 msgr2=0x7f39500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:27.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.863+0000 7f3954da0700 1 --2- 172.21.15.184:0/2764883270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39500f51a0 0x7f39500f55c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f3944009fb0 tx=0x7f3944009b30).stop 2024-02-19T22:18:27.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.863+0000 7f3954da0700 1 -- 172.21.15.184:0/2764883270 shutdown_connections 2024-02-19T22:18:27.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.863+0000 7f3954da0700 1 --2- 172.21.15.184:0/2764883270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39500f51a0 0x7f39500f55c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:27.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.863+0000 7f3954da0700 1 -- 172.21.15.184:0/2764883270 >> 172.21.15.184:0/2764883270 conn(0x7f39500f09a0 msgr2=0x7f39500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:27.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.863+0000 7f3954da0700 1 -- 172.21.15.184:0/2764883270 shutdown_connections 2024-02-19T22:18:27.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.863+0000 7f3954da0700 1 -- 172.21.15.184:0/2764883270 wait complete. 2024-02-19T22:18:27.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.864+0000 7f3954da0700 1 Processor -- start 2024-02-19T22:18:27.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.864+0000 7f3954da0700 1 -- start start 2024-02-19T22:18:27.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.864+0000 7f3954da0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39500f51a0 0x7f39501065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:27.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.864+0000 7f3954da0700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3950106ad0 con 0x7f39500f51a0 2024-02-19T22:18:27.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.864+0000 7f394f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39500f51a0 0x7f39501065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:27.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.864+0000 7f394f7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39500f51a0 0x7f39501065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:56712/0 (socket says 172.21.15.184:56712) 2024-02-19T22:18:27.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.864+0000 7f394f7fe700 1 -- 172.21.15.184:0/3326014963 learned_addr learned my addr 172.21.15.184:0/3326014963 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:27.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.865+0000 7f394f7fe700 1 -- 172.21.15.184:0/3326014963 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3944005040 con 0x7f39500f51a0 2024-02-19T22:18:27.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.865+0000 7f394f7fe700 1 --2- 172.21.15.184:0/3326014963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39500f51a0 0x7f39501065e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f39440089f0 tx=0x7f394400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:27.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.865+0000 7f3937fff700 1 -- 172.21.15.184:0/3326014963 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3944009320 con 0x7f39500f51a0 2024-02-19T22:18:27.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.865+0000 7f3954da0700 1 -- 172.21.15.184:0/3326014963 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3950106cd0 con 0x7f39500f51a0 2024-02-19T22:18:27.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.865+0000 7f3937fff700 1 -- 172.21.15.184:0/3326014963 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3944013380 con 0x7f39500f51a0 2024-02-19T22:18:27.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.865+0000 7f3937fff700 1 -- 172.21.15.184:0/3326014963 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3944014e20 con 0x7f39500f51a0 2024-02-19T22:18:27.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.865+0000 7f3954da0700 1 -- 172.21.15.184:0/3326014963 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3950107160 con 0x7f39500f51a0 2024-02-19T22:18:27.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.866+0000 7f3937fff700 1 -- 172.21.15.184:0/3326014963 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39440263a0 con 0x7f39500f51a0 2024-02-19T22:18:27.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.866+0000 7f3937fff700 1 --2- 172.21.15.184:0/3326014963 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f393802ff60 0x7f3938032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:27.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.866+0000 7f3937fff700 1 -- 172.21.15.184:0/3326014963 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3944041060 con 0x7f39500f51a0 2024-02-19T22:18:27.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.866+0000 7f3954da0700 1 -- 172.21.15.184:0/3326014963 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3950059100 con 0x7f39500f51a0 2024-02-19T22:18:27.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.866+0000 7f394effd700 1 --2- 172.21.15.184:0/3326014963 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f393802ff60 0x7f3938032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:27.869 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.867+0000 7f394effd700 1 --2- 172.21.15.184:0/3326014963 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f393802ff60 0x7f3938032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f39400096a0 tx=0x7f3940006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:27.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:27.871+0000 7f3937fff700 1 -- 172.21.15.184:0/3326014963 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3944012920 con 0x7f39500f51a0 2024-02-19T22:18:28.076 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:28.074+0000 7f3954da0700 1 -- 172.21.15.184:0/3326014963 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39500fad30 con 0x7f39500f51a0 2024-02-19T22:18:28.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:28.075+0000 7f3937fff700 1 -- 172.21.15.184:0/3326014963 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f394402b020 con 0x7f39500f51a0 2024-02-19T22:18:28.077 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:28.077 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:28.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:28.077+0000 7f3954da0700 1 -- 172.21.15.184:0/3326014963 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f393802ff60 msgr2=0x7f3938032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:28.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:28.077+0000 7f3954da0700 1 --2- 172.21.15.184:0/3326014963 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f393802ff60 0x7f3938032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f39400096a0 tx=0x7f3940006b40).stop 2024-02-19T22:18:28.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:28.077+0000 7f3954da0700 1 -- 172.21.15.184:0/3326014963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39500f51a0 msgr2=0x7f39501065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:28.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:28.077+0000 7f3954da0700 1 --2- 172.21.15.184:0/3326014963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39500f51a0 0x7f39501065e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f39440089f0 tx=0x7f394400b040).stop 2024-02-19T22:18:28.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:28.077+0000 7f3954da0700 1 -- 172.21.15.184:0/3326014963 shutdown_connections 2024-02-19T22:18:28.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:28.077+0000 7f3954da0700 1 --2- 172.21.15.184:0/3326014963 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f393802ff60 0x7f3938032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:28.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:28.077+0000 7f3954da0700 1 --2- 172.21.15.184:0/3326014963 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39500f51a0 0x7f39501065e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:28.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:28.077+0000 7f3954da0700 1 -- 172.21.15.184:0/3326014963 >> 172.21.15.184:0/3326014963 conn(0x7f39500f09a0 msgr2=0x7f39500f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:28.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:28.077+0000 7f3954da0700 1 -- 172.21.15.184:0/3326014963 shutdown_connections 2024-02-19T22:18:28.079 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:28.077+0000 7f3954da0700 1 -- 172.21.15.184:0/3326014963 wait complete. 2024-02-19T22:18:28.080 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:28.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:25.922583+0000 mgr.smithi176.faoogw (mgr.14162) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:29.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:28 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:28.076609+0000 mon.smithi176 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.184:0/3326014963' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:29.868 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:29.868 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:30.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:29 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:27.922915+0000 mgr.smithi176.faoogw (mgr.14162) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:31.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.689+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/3548858608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c780f5410 msgr2=0x7f4c780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:31.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.689+0000 7f4c7cfcd700 1 --2- 172.21.15.184:0/3548858608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c780f5410 0x7f4c780f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f4c68009fb0 tx=0x7f4c68009b30).stop 2024-02-19T22:18:31.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.689+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/3548858608 shutdown_connections 2024-02-19T22:18:31.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.689+0000 7f4c7cfcd700 1 --2- 172.21.15.184:0/3548858608 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c780f5410 0x7f4c780f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:31.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.689+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/3548858608 >> 172.21.15.184:0/3548858608 conn(0x7f4c780f0b70 msgr2=0x7f4c780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:31.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.689+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/3548858608 shutdown_connections 2024-02-19T22:18:31.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.689+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/3548858608 wait complete. 2024-02-19T22:18:31.694 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.689+0000 7f4c7cfcd700 1 Processor -- start 2024-02-19T22:18:31.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.689+0000 7f4c7cfcd700 1 -- start start 2024-02-19T22:18:31.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.690+0000 7f4c7cfcd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c780f5410 0x7f4c781067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:31.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.690+0000 7f4c7cfcd700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c78106cd0 con 0x7f4c780f5410 2024-02-19T22:18:31.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.690+0000 7f4c7659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c780f5410 0x7f4c781067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:31.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.690+0000 7f4c7659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c780f5410 0x7f4c781067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:56720/0 (socket says 172.21.15.184:56720) 2024-02-19T22:18:31.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.690+0000 7f4c7659c700 1 -- 172.21.15.184:0/1263530610 learned_addr learned my addr 172.21.15.184:0/1263530610 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:31.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.690+0000 7f4c7659c700 1 -- 172.21.15.184:0/1263530610 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c68005040 con 0x7f4c780f5410 2024-02-19T22:18:31.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.690+0000 7f4c7659c700 1 --2- 172.21.15.184:0/1263530610 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c780f5410 0x7f4c781067e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f4c68005010 tx=0x7f4c6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:31.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.690+0000 7f4c66ffd700 1 -- 172.21.15.184:0/1263530610 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c68009320 con 0x7f4c780f5410 2024-02-19T22:18:31.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.690+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/1263530610 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c78106ed0 con 0x7f4c780f5410 2024-02-19T22:18:31.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.690+0000 7f4c66ffd700 1 -- 172.21.15.184:0/1263530610 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c68013380 con 0x7f4c780f5410 2024-02-19T22:18:31.695 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.690+0000 7f4c66ffd700 1 -- 172.21.15.184:0/1263530610 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c68014e50 con 0x7f4c780f5410 2024-02-19T22:18:31.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.690+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/1263530610 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c78107360 con 0x7f4c780f5410 2024-02-19T22:18:31.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.691+0000 7f4c66ffd700 1 -- 172.21.15.184:0/1263530610 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c680095f0 con 0x7f4c780f5410 2024-02-19T22:18:31.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.691+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/1263530610 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c78045ad0 con 0x7f4c780f5410 2024-02-19T22:18:31.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.691+0000 7f4c66ffd700 1 --2- 172.21.15.184:0/1263530610 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f4c6002ff10 0x7f4c600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:31.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.691+0000 7f4c66ffd700 1 -- 172.21.15.184:0/1263530610 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c68040e70 con 0x7f4c780f5410 2024-02-19T22:18:31.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.691+0000 7f4c75d9b700 1 --2- 172.21.15.184:0/1263530610 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f4c6002ff10 0x7f4c600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:31.696 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.692+0000 7f4c75d9b700 1 --2- 172.21.15.184:0/1263530610 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f4c6002ff10 0x7f4c600323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4c6c0096a0 tx=0x7f4c6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:31.698 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.696+0000 7f4c66ffd700 1 -- 172.21.15.184:0/1263530610 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c68014420 con 0x7f4c780f5410 2024-02-19T22:18:31.910 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.908+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/1263530610 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c78108210 con 0x7f4c780f5410 2024-02-19T22:18:31.910 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.908+0000 7f4c66ffd700 1 -- 172.21.15.184:0/1263530610 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4c68010230 con 0x7f4c780f5410 2024-02-19T22:18:31.910 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:31.911 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:31.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.910+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/1263530610 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f4c6002ff10 msgr2=0x7f4c600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:31.912 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.910+0000 7f4c7cfcd700 1 --2- 172.21.15.184:0/1263530610 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f4c6002ff10 0x7f4c600323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4c6c0096a0 tx=0x7f4c6c006b40).stop 2024-02-19T22:18:31.913 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.910+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/1263530610 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c780f5410 msgr2=0x7f4c781067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:31.913 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.910+0000 7f4c7cfcd700 1 --2- 172.21.15.184:0/1263530610 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c780f5410 0x7f4c781067e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f4c68005010 tx=0x7f4c6800b040).stop 2024-02-19T22:18:31.913 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.910+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/1263530610 shutdown_connections 2024-02-19T22:18:31.913 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.910+0000 7f4c7cfcd700 1 --2- 172.21.15.184:0/1263530610 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f4c6002ff10 0x7f4c600323d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:31.913 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.910+0000 7f4c7cfcd700 1 --2- 172.21.15.184:0/1263530610 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4c780f5410 0x7f4c781067e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:31.913 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.910+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/1263530610 >> 172.21.15.184:0/1263530610 conn(0x7f4c780f0b70 msgr2=0x7f4c780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:31.913 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.910+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/1263530610 shutdown_connections 2024-02-19T22:18:31.913 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:31.910+0000 7f4c7cfcd700 1 -- 172.21.15.184:0/1263530610 wait complete. 2024-02-19T22:18:31.913 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:32.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:29.923269+0000 mgr.smithi176.faoogw (mgr.14162) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:33.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:32 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:31.909779+0000 mon.smithi176 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.184:0/1263530610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:33.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:32 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:32.511088+0000 mon.smithi176 (mon.0) 293 : audit [DBG] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:18:33.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:32 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:32.734529+0000 mon.smithi176 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:18:33.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:32 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:32.740871+0000 mon.smithi176 (mon.0) 295 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:18:33.648 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:33.649 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:34.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:31.923587+0000 mgr.smithi176.faoogw (mgr.14162) 306 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:34.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:18:32.510438+0000 mgr.smithi176.faoogw (mgr.14162) 307 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi176 ... 2024-02-19T22:18:34.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-19T22:18:34.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-19T22:18:34.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:16:39Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:34.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:17:17Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:34.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:17:55Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:34.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:18:34.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Traceback (most recent call last): 2024-02-19T22:18:34.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:18:34.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: main() 2024-02-19T22:18:34.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:18:34.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: r = ctx.func(ctx) 2024-02-19T22:18:34.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:18:34.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: return func(ctx) 2024-02-19T22:18:34.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:18:34.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:18:34.137 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-19T22:18:34.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-19T22:18:34.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:18:34.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: raise RuntimeError('uid/gid not found') 2024-02-19T22:18:34.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: RuntimeError: uid/gid not found 2024-02-19T22:18:34.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Traceback (most recent call last): 2024-02-19T22:18:34.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:18:34.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: yield (conn, connr) 2024-02-19T22:18:34.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:18:34.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: code, '\n'.join(err))) 2024-02-19T22:18:34.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi176 ... 2024-02-19T22:18:34.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi176 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-19T22:18:34.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-19T22:18:34.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:16:39Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:34.138 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:17:17Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr time="2024-02-19T22:17:55Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: Traceback (most recent call last): 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: main() 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: r = ctx.func(ctx) 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: return func(ctx) 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: File "/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:18:34.139 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: raise RuntimeError('uid/gid not found') 2024-02-19T22:18:34.140 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: RuntimeError: uid/gid not found 2024-02-19T22:18:34.140 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:32.965575+0000 mon.smithi176 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:18:35.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:33.923929+0000 mgr.smithi176.faoogw (mgr.14162) 308 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:35.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:34.617113+0000 mon.smithi176 (mon.0) 297 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:18:36.381 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.377+0000 7f8a2f7df700 1 -- 172.21.15.184:0/3128289625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a280f5410 msgr2=0x7f8a280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:36.381 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.377+0000 7f8a2f7df700 1 --2- 172.21.15.184:0/3128289625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a280f5410 0x7f8a280f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f8a18009fb0 tx=0x7f8a18009b30).stop 2024-02-19T22:18:36.381 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.377+0000 7f8a2f7df700 1 -- 172.21.15.184:0/3128289625 shutdown_connections 2024-02-19T22:18:36.381 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.377+0000 7f8a2f7df700 1 --2- 172.21.15.184:0/3128289625 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a280f5410 0x7f8a280f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:36.381 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.377+0000 7f8a2f7df700 1 -- 172.21.15.184:0/3128289625 >> 172.21.15.184:0/3128289625 conn(0x7f8a280f0b70 msgr2=0x7f8a280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:36.381 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.377+0000 7f8a2f7df700 1 -- 172.21.15.184:0/3128289625 shutdown_connections 2024-02-19T22:18:36.381 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.377+0000 7f8a2f7df700 1 -- 172.21.15.184:0/3128289625 wait complete. 2024-02-19T22:18:36.381 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.378+0000 7f8a2f7df700 1 Processor -- start 2024-02-19T22:18:36.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.378+0000 7f8a2f7df700 1 -- start start 2024-02-19T22:18:36.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.378+0000 7f8a2f7df700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a280f5410 0x7f8a281056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:36.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.378+0000 7f8a2f7df700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a28105bd0 con 0x7f8a280f5410 2024-02-19T22:18:36.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.378+0000 7f8a2d57b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a280f5410 0x7f8a281056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:36.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.378+0000 7f8a2d57b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a280f5410 0x7f8a281056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:38204/0 (socket says 172.21.15.184:38204) 2024-02-19T22:18:36.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.378+0000 7f8a2d57b700 1 -- 172.21.15.184:0/2088775943 learned_addr learned my addr 172.21.15.184:0/2088775943 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:36.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.378+0000 7f8a2d57b700 1 -- 172.21.15.184:0/2088775943 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a18005040 con 0x7f8a280f5410 2024-02-19T22:18:36.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.379+0000 7f8a2d57b700 1 --2- 172.21.15.184:0/2088775943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a280f5410 0x7f8a281056e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f8a18005010 tx=0x7f8a1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:36.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.379+0000 7f8a1dffb700 1 -- 172.21.15.184:0/2088775943 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a18009320 con 0x7f8a280f5410 2024-02-19T22:18:36.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.379+0000 7f8a2f7df700 1 -- 172.21.15.184:0/2088775943 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a28105dd0 con 0x7f8a280f5410 2024-02-19T22:18:36.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.379+0000 7f8a1dffb700 1 -- 172.21.15.184:0/2088775943 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a18013380 con 0x7f8a280f5410 2024-02-19T22:18:36.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.379+0000 7f8a1dffb700 1 -- 172.21.15.184:0/2088775943 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a18014e50 con 0x7f8a280f5410 2024-02-19T22:18:36.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.379+0000 7f8a2f7df700 1 -- 172.21.15.184:0/2088775943 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a28106260 con 0x7f8a280f5410 2024-02-19T22:18:36.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.380+0000 7f8a1dffb700 1 -- 172.21.15.184:0/2088775943 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a180263a0 con 0x7f8a280f5410 2024-02-19T22:18:36.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.380+0000 7f8a1dffb700 1 --2- 172.21.15.184:0/2088775943 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8a1402ff10 0x7f8a140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:36.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.380+0000 7f8a1dffb700 1 -- 172.21.15.184:0/2088775943 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a18040f40 con 0x7f8a280f5410 2024-02-19T22:18:36.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.380+0000 7f8a2f7df700 1 -- 172.21.15.184:0/2088775943 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a28045ad0 con 0x7f8a280f5410 2024-02-19T22:18:36.384 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.380+0000 7f8a2cd7a700 1 --2- 172.21.15.184:0/2088775943 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8a1402ff10 0x7f8a140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:36.384 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.381+0000 7f8a2cd7a700 1 --2- 172.21.15.184:0/2088775943 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8a1402ff10 0x7f8a140323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f8a240096a0 tx=0x7f8a24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:36.386 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.385+0000 7f8a1dffb700 1 -- 172.21.15.184:0/2088775943 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a18012920 con 0x7f8a280f5410 2024-02-19T22:18:36.606 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.604+0000 7f8a2f7df700 1 -- 172.21.15.184:0/2088775943 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a28106fa0 con 0x7f8a280f5410 2024-02-19T22:18:36.606 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.604+0000 7f8a1dffb700 1 -- 172.21.15.184:0/2088775943 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8a1802b020 con 0x7f8a280f5410 2024-02-19T22:18:36.607 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:36.607 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:36.608 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.606+0000 7f8a2f7df700 1 -- 172.21.15.184:0/2088775943 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8a1402ff10 msgr2=0x7f8a140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:36.608 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.606+0000 7f8a2f7df700 1 --2- 172.21.15.184:0/2088775943 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8a1402ff10 0x7f8a140323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f8a240096a0 tx=0x7f8a24006b40).stop 2024-02-19T22:18:36.608 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.606+0000 7f8a2f7df700 1 -- 172.21.15.184:0/2088775943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a280f5410 msgr2=0x7f8a281056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:36.608 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.606+0000 7f8a2f7df700 1 --2- 172.21.15.184:0/2088775943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a280f5410 0x7f8a281056e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f8a18005010 tx=0x7f8a1800b040).stop 2024-02-19T22:18:36.609 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.606+0000 7f8a2f7df700 1 -- 172.21.15.184:0/2088775943 shutdown_connections 2024-02-19T22:18:36.609 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.606+0000 7f8a2f7df700 1 --2- 172.21.15.184:0/2088775943 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8a1402ff10 0x7f8a140323d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:36.609 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.606+0000 7f8a2f7df700 1 --2- 172.21.15.184:0/2088775943 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a280f5410 0x7f8a281056e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:36.609 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.606+0000 7f8a2f7df700 1 -- 172.21.15.184:0/2088775943 >> 172.21.15.184:0/2088775943 conn(0x7f8a280f0b70 msgr2=0x7f8a280f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:36.609 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.606+0000 7f8a2f7df700 1 -- 172.21.15.184:0/2088775943 shutdown_connections 2024-02-19T22:18:36.609 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:36.606+0000 7f8a2f7df700 1 -- 172.21.15.184:0/2088775943 wait complete. 2024-02-19T22:18:36.611 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:36.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:35.975939+0000 mon.smithi176 (mon.0) 298 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:18:36.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:35.977262+0000 mon.smithi176 (mon.0) 299 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:18:36.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:36.605752+0000 mon.smithi176 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.184:0/2088775943' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:37.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:35.924234+0000 mgr.smithi176.faoogw (mgr.14162) 309 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:38.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:38.562 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:39.386 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:37.924617+0000 mgr.smithi176.faoogw (mgr.14162) 310 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:39.386 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:38.120164+0000 mon.smithi176 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:18:39.386 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:38.392705+0000 mon.smithi176 (mon.0) 302 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:18:39.386 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:38.393404+0000 mgr.smithi176.faoogw (mgr.14162) 311 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:39.386 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:38.395490+0000 mon.smithi176 (mon.0) 303 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' 2024-02-19T22:18:39.386 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cephadm 2024-02-19T22:18:38.397755+0000 mgr.smithi176.faoogw (mgr.14162) 312 : cephadm [INF] Deploying daemon alertmanager.smithi184 on smithi184 2024-02-19T22:18:39.386 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:38.620580+0000 mon.smithi176 (mon.0) 304 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-19T22:18:40.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.404+0000 7f66f75a7700 1 -- 172.21.15.184:0/2266711202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66f00f4740 msgr2=0x7f66f00f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:40.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.404+0000 7f66f75a7700 1 --2- 172.21.15.184:0/2266711202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66f00f4740 0x7f66f00f4b60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f66e0009fb0 tx=0x7f66e0009b30).stop 2024-02-19T22:18:40.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.404+0000 7f66f75a7700 1 -- 172.21.15.184:0/2266711202 shutdown_connections 2024-02-19T22:18:40.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.404+0000 7f66f75a7700 1 --2- 172.21.15.184:0/2266711202 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66f00f4740 0x7f66f00f4b60 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:40.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.404+0000 7f66f75a7700 1 -- 172.21.15.184:0/2266711202 >> 172.21.15.184:0/2266711202 conn(0x7f66f00f0260 msgr2=0x7f66f00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:40.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.404+0000 7f66f75a7700 1 -- 172.21.15.184:0/2266711202 shutdown_connections 2024-02-19T22:18:40.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.404+0000 7f66f75a7700 1 -- 172.21.15.184:0/2266711202 wait complete. 2024-02-19T22:18:40.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.404+0000 7f66f75a7700 1 Processor -- start 2024-02-19T22:18:40.408 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.404+0000 7f66f75a7700 1 -- start start 2024-02-19T22:18:40.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.405+0000 7f66f75a7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66f00f4740 0x7f66f01067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:40.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.405+0000 7f66f75a7700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f0106ca0 con 0x7f66f00f4740 2024-02-19T22:18:40.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.405+0000 7f66f5343700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66f00f4740 0x7f66f01067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:40.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.405+0000 7f66f5343700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66f00f4740 0x7f66f01067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:38228/0 (socket says 172.21.15.184:38228) 2024-02-19T22:18:40.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.405+0000 7f66f5343700 1 -- 172.21.15.184:0/3807671237 learned_addr learned my addr 172.21.15.184:0/3807671237 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:40.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.405+0000 7f66f5343700 1 -- 172.21.15.184:0/3807671237 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e0005040 con 0x7f66f00f4740 2024-02-19T22:18:40.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.405+0000 7f66f5343700 1 --2- 172.21.15.184:0/3807671237 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66f00f4740 0x7f66f01067b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f66e0005010 tx=0x7f66e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:40.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.406+0000 7f66e5ffb700 1 -- 172.21.15.184:0/3807671237 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66e0009320 con 0x7f66f00f4740 2024-02-19T22:18:40.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.406+0000 7f66f75a7700 1 -- 172.21.15.184:0/3807671237 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f0106ea0 con 0x7f66f00f4740 2024-02-19T22:18:40.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.406+0000 7f66e5ffb700 1 -- 172.21.15.184:0/3807671237 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66e0013380 con 0x7f66f00f4740 2024-02-19T22:18:40.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.406+0000 7f66e5ffb700 1 -- 172.21.15.184:0/3807671237 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66e0014e50 con 0x7f66f00f4740 2024-02-19T22:18:40.409 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.406+0000 7f66f75a7700 1 -- 172.21.15.184:0/3807671237 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f0107330 con 0x7f66f00f4740 2024-02-19T22:18:40.410 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.407+0000 7f66e5ffb700 1 -- 172.21.15.184:0/3807671237 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66e00095f0 con 0x7f66f00f4740 2024-02-19T22:18:40.410 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.407+0000 7f66f75a7700 1 -- 172.21.15.184:0/3807671237 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f0045ad0 con 0x7f66f00f4740 2024-02-19T22:18:40.411 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.407+0000 7f66e5ffb700 1 --2- 172.21.15.184:0/3807671237 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f66dc02ff10 0x7f66dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:40.411 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.407+0000 7f66e5ffb700 1 -- 172.21.15.184:0/3807671237 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66e0040fa0 con 0x7f66f00f4740 2024-02-19T22:18:40.411 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.407+0000 7f66f4b42700 1 --2- 172.21.15.184:0/3807671237 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f66dc02ff10 0x7f66dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:40.411 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.407+0000 7f66f4b42700 1 --2- 172.21.15.184:0/3807671237 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f66dc02ff10 0x7f66dc0323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f66ec0096a0 tx=0x7f66ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:40.413 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.411+0000 7f66e5ffb700 1 -- 172.21.15.184:0/3807671237 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66e0014420 con 0x7f66f00f4740 2024-02-19T22:18:40.620 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.618+0000 7f66f75a7700 1 -- 172.21.15.184:0/3807671237 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66f01081e0 con 0x7f66f00f4740 2024-02-19T22:18:40.621 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.619+0000 7f66e5ffb700 1 -- 172.21.15.184:0/3807671237 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f66e0010230 con 0x7f66f00f4740 2024-02-19T22:18:40.621 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:40.622 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:40.623 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.621+0000 7f66f75a7700 1 -- 172.21.15.184:0/3807671237 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f66dc02ff10 msgr2=0x7f66dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:40.623 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.621+0000 7f66f75a7700 1 --2- 172.21.15.184:0/3807671237 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f66dc02ff10 0x7f66dc0323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f66ec0096a0 tx=0x7f66ec006b40).stop 2024-02-19T22:18:40.623 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.621+0000 7f66f75a7700 1 -- 172.21.15.184:0/3807671237 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66f00f4740 msgr2=0x7f66f01067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:40.623 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.621+0000 7f66f75a7700 1 --2- 172.21.15.184:0/3807671237 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66f00f4740 0x7f66f01067b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f66e0005010 tx=0x7f66e000b040).stop 2024-02-19T22:18:40.623 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.621+0000 7f66f75a7700 1 -- 172.21.15.184:0/3807671237 shutdown_connections 2024-02-19T22:18:40.623 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.621+0000 7f66f75a7700 1 --2- 172.21.15.184:0/3807671237 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f66dc02ff10 0x7f66dc0323d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:40.623 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.621+0000 7f66f75a7700 1 --2- 172.21.15.184:0/3807671237 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f66f00f4740 0x7f66f01067b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:40.624 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.621+0000 7f66f75a7700 1 -- 172.21.15.184:0/3807671237 >> 172.21.15.184:0/3807671237 conn(0x7f66f00f0260 msgr2=0x7f66f00f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:40.624 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.621+0000 7f66f75a7700 1 -- 172.21.15.184:0/3807671237 shutdown_connections 2024-02-19T22:18:40.624 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:40.621+0000 7f66f75a7700 1 -- 172.21.15.184:0/3807671237 wait complete. 2024-02-19T22:18:40.624 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:41.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:40 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:40.620428+0000 mon.smithi176 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.184:0/3807671237' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:42.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:40.393788+0000 mgr.smithi176.faoogw (mgr.14162) 313 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:42.447 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:42.447 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:44.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:42.394183+0000 mgr.smithi176.faoogw (mgr.14162) 314 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:44.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.305+0000 7f3312dd0700 1 -- 172.21.15.184:0/3708007326 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0f5410 msgr2=0x7f330c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:44.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.305+0000 7f3312dd0700 1 --2- 172.21.15.184:0/3708007326 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0f5410 0x7f330c0f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f3300009fb0 tx=0x7f3300009b30).stop 2024-02-19T22:18:44.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.305+0000 7f3312dd0700 1 -- 172.21.15.184:0/3708007326 shutdown_connections 2024-02-19T22:18:44.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.305+0000 7f3312dd0700 1 --2- 172.21.15.184:0/3708007326 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0f5410 0x7f330c0f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:44.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.305+0000 7f3312dd0700 1 -- 172.21.15.184:0/3708007326 >> 172.21.15.184:0/3708007326 conn(0x7f330c0f0b70 msgr2=0x7f330c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:44.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.305+0000 7f3312dd0700 1 -- 172.21.15.184:0/3708007326 shutdown_connections 2024-02-19T22:18:44.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.305+0000 7f3312dd0700 1 -- 172.21.15.184:0/3708007326 wait complete. 2024-02-19T22:18:44.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.306+0000 7f3312dd0700 1 Processor -- start 2024-02-19T22:18:44.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.306+0000 7f3312dd0700 1 -- start start 2024-02-19T22:18:44.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.306+0000 7f3312dd0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0f5410 0x7f330c105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:44.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.307+0000 7f3312dd0700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c105c00 con 0x7f330c0f5410 2024-02-19T22:18:44.346 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.307+0000 7f3310b6c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0f5410 0x7f330c105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:44.347 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.307+0000 7f3310b6c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0f5410 0x7f330c105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:38248/0 (socket says 172.21.15.184:38248) 2024-02-19T22:18:44.347 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.307+0000 7f3310b6c700 1 -- 172.21.15.184:0/1390998374 learned_addr learned my addr 172.21.15.184:0/1390998374 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:44.347 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.307+0000 7f3310b6c700 1 -- 172.21.15.184:0/1390998374 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3300005040 con 0x7f330c0f5410 2024-02-19T22:18:44.347 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.307+0000 7f3310b6c700 1 --2- 172.21.15.184:0/1390998374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0f5410 0x7f330c105710 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f3300005010 tx=0x7f330000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:44.347 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.307+0000 7f33097fa700 1 -- 172.21.15.184:0/1390998374 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3300009320 con 0x7f330c0f5410 2024-02-19T22:18:44.347 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.307+0000 7f33097fa700 1 -- 172.21.15.184:0/1390998374 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3300013380 con 0x7f330c0f5410 2024-02-19T22:18:44.347 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.307+0000 7f3312dd0700 1 -- 172.21.15.184:0/1390998374 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f330c105e00 con 0x7f330c0f5410 2024-02-19T22:18:44.347 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.308+0000 7f33097fa700 1 -- 172.21.15.184:0/1390998374 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3300014e50 con 0x7f330c0f5410 2024-02-19T22:18:44.347 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.308+0000 7f3312dd0700 1 -- 172.21.15.184:0/1390998374 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330c106290 con 0x7f330c0f5410 2024-02-19T22:18:44.347 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.308+0000 7f33097fa700 1 -- 172.21.15.184:0/1390998374 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33000095f0 con 0x7f330c0f5410 2024-02-19T22:18:44.347 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.308+0000 7f3312dd0700 1 -- 172.21.15.184:0/1390998374 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330c045ad0 con 0x7f330c0f5410 2024-02-19T22:18:44.347 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.308+0000 7f33097fa700 1 --2- 172.21.15.184:0/1390998374 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f32fc02ff10 0x7f32fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:44.347 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.308+0000 7f33097fa700 1 -- 172.21.15.184:0/1390998374 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3300040fa0 con 0x7f330c0f5410 2024-02-19T22:18:44.348 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.309+0000 7f330bfff700 1 --2- 172.21.15.184:0/1390998374 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f32fc02ff10 0x7f32fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:44.348 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.309+0000 7f330bfff700 1 --2- 172.21.15.184:0/1390998374 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f32fc02ff10 0x7f32fc0323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f32f80096a0 tx=0x7f32f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:44.348 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.313+0000 7f33097fa700 1 -- 172.21.15.184:0/1390998374 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3300014420 con 0x7f330c0f5410 2024-02-19T22:18:44.519 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.517+0000 7f3312dd0700 1 -- 172.21.15.184:0/1390998374 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f330c107140 con 0x7f330c0f5410 2024-02-19T22:18:44.520 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.518+0000 7f33097fa700 1 -- 172.21.15.184:0/1390998374 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3300010230 con 0x7f330c0f5410 2024-02-19T22:18:44.520 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:44.520 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:44.521 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.519+0000 7f3312dd0700 1 -- 172.21.15.184:0/1390998374 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f32fc02ff10 msgr2=0x7f32fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:44.521 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.519+0000 7f3312dd0700 1 --2- 172.21.15.184:0/1390998374 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f32fc02ff10 0x7f32fc0323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f32f80096a0 tx=0x7f32f8006b40).stop 2024-02-19T22:18:44.521 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.519+0000 7f3312dd0700 1 -- 172.21.15.184:0/1390998374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0f5410 msgr2=0x7f330c105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:44.521 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.519+0000 7f3312dd0700 1 --2- 172.21.15.184:0/1390998374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0f5410 0x7f330c105710 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f3300005010 tx=0x7f330000b040).stop 2024-02-19T22:18:44.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.519+0000 7f3312dd0700 1 -- 172.21.15.184:0/1390998374 shutdown_connections 2024-02-19T22:18:44.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.519+0000 7f3312dd0700 1 --2- 172.21.15.184:0/1390998374 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f32fc02ff10 0x7f32fc0323d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:44.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.519+0000 7f3312dd0700 1 --2- 172.21.15.184:0/1390998374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f330c0f5410 0x7f330c105710 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:44.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.519+0000 7f3312dd0700 1 -- 172.21.15.184:0/1390998374 >> 172.21.15.184:0/1390998374 conn(0x7f330c0f0b70 msgr2=0x7f330c0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:44.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.519+0000 7f3312dd0700 1 -- 172.21.15.184:0/1390998374 shutdown_connections 2024-02-19T22:18:44.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:44.519+0000 7f3312dd0700 1 -- 172.21.15.184:0/1390998374 wait complete. 2024-02-19T22:18:44.522 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:45.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:44 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:44.519010+0000 mon.smithi176 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.184:0/1390998374' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:46.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:44.394551+0000 mgr.smithi176.faoogw (mgr.14162) 315 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:46.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:46.279 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:48.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:46.394902+0000 mgr.smithi176.faoogw (mgr.14162) 316 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:48.138 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.135+0000 7f21fcae0700 1 -- 172.21.15.184:0/1289477769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21f80f6510 msgr2=0x7f21f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:48.138 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.135+0000 7f21fcae0700 1 --2- 172.21.15.184:0/1289477769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21f80f6510 0x7f21f80f6930 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f21ec009fb0 tx=0x7f21ec009b30).stop 2024-02-19T22:18:48.138 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.135+0000 7f21fcae0700 1 -- 172.21.15.184:0/1289477769 shutdown_connections 2024-02-19T22:18:48.138 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.135+0000 7f21fcae0700 1 --2- 172.21.15.184:0/1289477769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21f80f6510 0x7f21f80f6930 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.135+0000 7f21fcae0700 1 -- 172.21.15.184:0/1289477769 >> 172.21.15.184:0/1289477769 conn(0x7f21f80f1c70 msgr2=0x7f21f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.135+0000 7f21fcae0700 1 -- 172.21.15.184:0/1289477769 shutdown_connections 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.135+0000 7f21fcae0700 1 -- 172.21.15.184:0/1289477769 wait complete. 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.135+0000 7f21fcae0700 1 Processor -- start 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.135+0000 7f21fcae0700 1 -- start start 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.136+0000 7f21fcae0700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21f80f6510 0x7f21f8106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.136+0000 7f21fcae0700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f8106d00 con 0x7f21f80f6510 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.136+0000 7f21f659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21f80f6510 0x7f21f8106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.136+0000 7f21f659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21f80f6510 0x7f21f8106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:59002/0 (socket says 172.21.15.184:59002) 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.136+0000 7f21f659c700 1 -- 172.21.15.184:0/3907003704 learned_addr learned my addr 172.21.15.184:0/3907003704 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.136+0000 7f21f659c700 1 -- 172.21.15.184:0/3907003704 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ec005040 con 0x7f21f80f6510 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.136+0000 7f21f659c700 1 --2- 172.21.15.184:0/3907003704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21f80f6510 0x7f21f8106810 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f21ec005010 tx=0x7f21ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.136+0000 7f21e6ffd700 1 -- 172.21.15.184:0/3907003704 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21ec009320 con 0x7f21f80f6510 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.136+0000 7f21fcae0700 1 -- 172.21.15.184:0/3907003704 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f8106f00 con 0x7f21f80f6510 2024-02-19T22:18:48.139 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.136+0000 7f21e6ffd700 1 -- 172.21.15.184:0/3907003704 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f21ec013380 con 0x7f21f80f6510 2024-02-19T22:18:48.140 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.136+0000 7f21e6ffd700 1 -- 172.21.15.184:0/3907003704 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21ec014e50 con 0x7f21f80f6510 2024-02-19T22:18:48.140 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.136+0000 7f21fcae0700 1 -- 172.21.15.184:0/3907003704 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f8107390 con 0x7f21f80f6510 2024-02-19T22:18:48.140 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.137+0000 7f21e6ffd700 1 -- 172.21.15.184:0/3907003704 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21ec0263a0 con 0x7f21f80f6510 2024-02-19T22:18:48.140 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.137+0000 7f21e6ffd700 1 --2- 172.21.15.184:0/3907003704 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f21e002ff10 0x7f21e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.137+0000 7f21e6ffd700 1 -- 172.21.15.184:0/3907003704 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f21ec040e90 con 0x7f21f80f6510 2024-02-19T22:18:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.137+0000 7f21fcae0700 1 -- 172.21.15.184:0/3907003704 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f8045ad0 con 0x7f21f80f6510 2024-02-19T22:18:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.138+0000 7f21f5d9b700 1 --2- 172.21.15.184:0/3907003704 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f21e002ff10 0x7f21e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:48.141 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.138+0000 7f21f5d9b700 1 --2- 172.21.15.184:0/3907003704 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f21e002ff10 0x7f21e00323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f21e80096a0 tx=0x7f21e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:48.144 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.142+0000 7f21e6ffd700 1 -- 172.21.15.184:0/3907003704 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21ec0672b0 con 0x7f21f80f6510 2024-02-19T22:18:48.349 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.347+0000 7f21fcae0700 1 -- 172.21.15.184:0/3907003704 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21f81080e0 con 0x7f21f80f6510 2024-02-19T22:18:48.350 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.348+0000 7f21e6ffd700 1 -- 172.21.15.184:0/3907003704 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f21ec012920 con 0x7f21f80f6510 2024-02-19T22:18:48.350 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:48.350 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:48.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.349+0000 7f21fcae0700 1 -- 172.21.15.184:0/3907003704 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f21e002ff10 msgr2=0x7f21e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:48.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.349+0000 7f21fcae0700 1 --2- 172.21.15.184:0/3907003704 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f21e002ff10 0x7f21e00323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f21e80096a0 tx=0x7f21e8006b40).stop 2024-02-19T22:18:48.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.349+0000 7f21fcae0700 1 -- 172.21.15.184:0/3907003704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21f80f6510 msgr2=0x7f21f8106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:48.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.349+0000 7f21fcae0700 1 --2- 172.21.15.184:0/3907003704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21f80f6510 0x7f21f8106810 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f21ec005010 tx=0x7f21ec00b040).stop 2024-02-19T22:18:48.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.350+0000 7f21fcae0700 1 -- 172.21.15.184:0/3907003704 shutdown_connections 2024-02-19T22:18:48.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.350+0000 7f21fcae0700 1 --2- 172.21.15.184:0/3907003704 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f21e002ff10 0x7f21e00323d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:48.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.350+0000 7f21fcae0700 1 --2- 172.21.15.184:0/3907003704 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f21f80f6510 0x7f21f8106810 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:48.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.350+0000 7f21fcae0700 1 -- 172.21.15.184:0/3907003704 >> 172.21.15.184:0/3907003704 conn(0x7f21f80f1c70 msgr2=0x7f21f80f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:48.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.350+0000 7f21fcae0700 1 -- 172.21.15.184:0/3907003704 shutdown_connections 2024-02-19T22:18:48.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:48.350+0000 7f21fcae0700 1 -- 172.21.15.184:0/3907003704 wait complete. 2024-02-19T22:18:48.353 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:49.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:48 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:48.349310+0000 mon.smithi176 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.184:0/3907003704' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:50.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:50.050 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:50.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:49 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:48.395243+0000 mgr.smithi176.faoogw (mgr.14162) 317 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:51.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.780+0000 7fee8bc3b700 1 -- 172.21.15.184:0/1310907416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee840f5410 msgr2=0x7fee840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:51.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.780+0000 7fee8bc3b700 1 --2- 172.21.15.184:0/1310907416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee840f5410 0x7fee840f5830 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fee74009fb0 tx=0x7fee74009b30).stop 2024-02-19T22:18:51.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.780+0000 7fee8bc3b700 1 -- 172.21.15.184:0/1310907416 shutdown_connections 2024-02-19T22:18:51.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.780+0000 7fee8bc3b700 1 --2- 172.21.15.184:0/1310907416 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee840f5410 0x7fee840f5830 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:51.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.780+0000 7fee8bc3b700 1 -- 172.21.15.184:0/1310907416 >> 172.21.15.184:0/1310907416 conn(0x7fee840f0b70 msgr2=0x7fee840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:51.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.780+0000 7fee8bc3b700 1 -- 172.21.15.184:0/1310907416 shutdown_connections 2024-02-19T22:18:51.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.780+0000 7fee8bc3b700 1 -- 172.21.15.184:0/1310907416 wait complete. 2024-02-19T22:18:51.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.781+0000 7fee8bc3b700 1 Processor -- start 2024-02-19T22:18:51.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.781+0000 7fee8bc3b700 1 -- start start 2024-02-19T22:18:51.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.781+0000 7fee8bc3b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee840f5410 0x7fee841067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:51.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.781+0000 7fee8bc3b700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee84106cd0 con 0x7fee840f5410 2024-02-19T22:18:51.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.781+0000 7fee899d7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee840f5410 0x7fee841067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:51.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.781+0000 7fee899d7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee840f5410 0x7fee841067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:59010/0 (socket says 172.21.15.184:59010) 2024-02-19T22:18:51.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.781+0000 7fee899d7700 1 -- 172.21.15.184:0/4034016325 learned_addr learned my addr 172.21.15.184:0/4034016325 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:51.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.782+0000 7fee899d7700 1 -- 172.21.15.184:0/4034016325 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee74005040 con 0x7fee840f5410 2024-02-19T22:18:51.800 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.782+0000 7fee899d7700 1 --2- 172.21.15.184:0/4034016325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee840f5410 0x7fee841067e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fee74005010 tx=0x7fee7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:51.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.782+0000 7fee7a7fc700 1 -- 172.21.15.184:0/4034016325 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee74009320 con 0x7fee840f5410 2024-02-19T22:18:51.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.782+0000 7fee8bc3b700 1 -- 172.21.15.184:0/4034016325 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee84106ed0 con 0x7fee840f5410 2024-02-19T22:18:51.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.782+0000 7fee7a7fc700 1 -- 172.21.15.184:0/4034016325 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee74013380 con 0x7fee840f5410 2024-02-19T22:18:51.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.782+0000 7fee7a7fc700 1 -- 172.21.15.184:0/4034016325 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee74014e50 con 0x7fee840f5410 2024-02-19T22:18:51.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.782+0000 7fee8bc3b700 1 -- 172.21.15.184:0/4034016325 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee84107360 con 0x7fee840f5410 2024-02-19T22:18:51.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.783+0000 7fee7a7fc700 1 -- 172.21.15.184:0/4034016325 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee740095f0 con 0x7fee840f5410 2024-02-19T22:18:51.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.783+0000 7fee8bc3b700 1 -- 172.21.15.184:0/4034016325 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee84045ad0 con 0x7fee840f5410 2024-02-19T22:18:51.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.783+0000 7fee7a7fc700 1 --2- 172.21.15.184:0/4034016325 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fee7002ff10 0x7fee700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:51.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.783+0000 7fee7a7fc700 1 -- 172.21.15.184:0/4034016325 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee74040e90 con 0x7fee840f5410 2024-02-19T22:18:51.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.783+0000 7fee891d6700 1 --2- 172.21.15.184:0/4034016325 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fee7002ff10 0x7fee700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:51.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.784+0000 7fee891d6700 1 --2- 172.21.15.184:0/4034016325 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fee7002ff10 0x7fee700323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fee800096a0 tx=0x7fee80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:51.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.788+0000 7fee7a7fc700 1 -- 172.21.15.184:0/4034016325 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee74014420 con 0x7fee840f5410 2024-02-19T22:18:51.996 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.994+0000 7fee8bc3b700 1 -- 172.21.15.184:0/4034016325 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee840fafa0 con 0x7fee840f5410 2024-02-19T22:18:51.997 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.995+0000 7fee7a7fc700 1 -- 172.21.15.184:0/4034016325 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fee74010230 con 0x7fee840f5410 2024-02-19T22:18:51.998 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:51.998 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:51.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.997+0000 7fee8bc3b700 1 -- 172.21.15.184:0/4034016325 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fee7002ff10 msgr2=0x7fee700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:51.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.997+0000 7fee8bc3b700 1 --2- 172.21.15.184:0/4034016325 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fee7002ff10 0x7fee700323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fee800096a0 tx=0x7fee80006b40).stop 2024-02-19T22:18:51.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.997+0000 7fee8bc3b700 1 -- 172.21.15.184:0/4034016325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee840f5410 msgr2=0x7fee841067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:51.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.997+0000 7fee8bc3b700 1 --2- 172.21.15.184:0/4034016325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee840f5410 0x7fee841067e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fee74005010 tx=0x7fee7400b040).stop 2024-02-19T22:18:51.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.997+0000 7fee8bc3b700 1 -- 172.21.15.184:0/4034016325 shutdown_connections 2024-02-19T22:18:51.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.997+0000 7fee8bc3b700 1 --2- 172.21.15.184:0/4034016325 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fee7002ff10 0x7fee700323d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:51.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.997+0000 7fee8bc3b700 1 --2- 172.21.15.184:0/4034016325 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fee840f5410 0x7fee841067e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:51.999 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.997+0000 7fee8bc3b700 1 -- 172.21.15.184:0/4034016325 >> 172.21.15.184:0/4034016325 conn(0x7fee840f0b70 msgr2=0x7fee8405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:52.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.997+0000 7fee8bc3b700 1 -- 172.21.15.184:0/4034016325 shutdown_connections 2024-02-19T22:18:52.000 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:51.997+0000 7fee8bc3b700 1 -- 172.21.15.184:0/4034016325 wait complete. 2024-02-19T22:18:52.000 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:52.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:51 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:50.395563+0000 mgr.smithi176.faoogw (mgr.14162) 318 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:53.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:52 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:51.996455+0000 mon.smithi176 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.184:0/4034016325' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:53.705 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:53.705 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:54.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:52.395849+0000 mgr.smithi176.faoogw (mgr.14162) 319 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:55.639 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.637+0000 7fb2216e1700 1 -- 172.21.15.184:0/3439866804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb21c0f5410 msgr2=0x7fb21c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:55.639 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.637+0000 7fb2216e1700 1 --2- 172.21.15.184:0/3439866804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb21c0f5410 0x7fb21c0f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fb210009fb0 tx=0x7fb210009b30).stop 2024-02-19T22:18:55.640 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.637+0000 7fb2216e1700 1 -- 172.21.15.184:0/3439866804 shutdown_connections 2024-02-19T22:18:55.640 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.637+0000 7fb2216e1700 1 --2- 172.21.15.184:0/3439866804 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb21c0f5410 0x7fb21c0f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:55.640 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.637+0000 7fb2216e1700 1 -- 172.21.15.184:0/3439866804 >> 172.21.15.184:0/3439866804 conn(0x7fb21c0f0b70 msgr2=0x7fb21c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:55.640 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.637+0000 7fb2216e1700 1 -- 172.21.15.184:0/3439866804 shutdown_connections 2024-02-19T22:18:55.640 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.637+0000 7fb2216e1700 1 -- 172.21.15.184:0/3439866804 wait complete. 2024-02-19T22:18:55.640 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.637+0000 7fb2216e1700 1 Processor -- start 2024-02-19T22:18:55.640 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.637+0000 7fb2216e1700 1 -- start start 2024-02-19T22:18:55.640 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.638+0000 7fb2216e1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb21c0f5410 0x7fb21c109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:55.640 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.638+0000 7fb2216e1700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c060220 con 0x7fb21c0f5410 2024-02-19T22:18:55.642 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.638+0000 7fb21affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb21c0f5410 0x7fb21c109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:55.642 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.638+0000 7fb21affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb21c0f5410 0x7fb21c109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:59024/0 (socket says 172.21.15.184:59024) 2024-02-19T22:18:55.642 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.638+0000 7fb21affd700 1 -- 172.21.15.184:0/881197108 learned_addr learned my addr 172.21.15.184:0/881197108 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:55.642 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.638+0000 7fb21affd700 1 -- 172.21.15.184:0/881197108 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb210005040 con 0x7fb21c0f5410 2024-02-19T22:18:55.642 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.638+0000 7fb21affd700 1 --2- 172.21.15.184:0/881197108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb21c0f5410 0x7fb21c109500 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fb210005010 tx=0x7fb21000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:55.642 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.638+0000 7fb2037fe700 1 -- 172.21.15.184:0/881197108 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb210009320 con 0x7fb21c0f5410 2024-02-19T22:18:55.642 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.638+0000 7fb2037fe700 1 -- 172.21.15.184:0/881197108 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb210013380 con 0x7fb21c0f5410 2024-02-19T22:18:55.642 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.638+0000 7fb2216e1700 1 -- 172.21.15.184:0/881197108 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21c060420 con 0x7fb21c0f5410 2024-02-19T22:18:55.642 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.638+0000 7fb2037fe700 1 -- 172.21.15.184:0/881197108 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb210014e50 con 0x7fb21c0f5410 2024-02-19T22:18:55.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.638+0000 7fb2216e1700 1 -- 172.21.15.184:0/881197108 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21c109bb0 con 0x7fb21c0f5410 2024-02-19T22:18:55.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.639+0000 7fb2037fe700 1 -- 172.21.15.184:0/881197108 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2100095f0 con 0x7fb21c0f5410 2024-02-19T22:18:55.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.639+0000 7fb2216e1700 1 -- 172.21.15.184:0/881197108 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21c059100 con 0x7fb21c0f5410 2024-02-19T22:18:55.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.639+0000 7fb2037fe700 1 --2- 172.21.15.184:0/881197108 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb20402ff10 0x7fb2040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:55.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.639+0000 7fb2037fe700 1 -- 172.21.15.184:0/881197108 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb210040ee0 con 0x7fb21c0f5410 2024-02-19T22:18:55.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.639+0000 7fb21a7fc700 1 --2- 172.21.15.184:0/881197108 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb20402ff10 0x7fb2040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:55.643 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.640+0000 7fb21a7fc700 1 --2- 172.21.15.184:0/881197108 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb20402ff10 0x7fb2040323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fb20c0096a0 tx=0x7fb20c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:55.647 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.644+0000 7fb2037fe700 1 -- 172.21.15.184:0/881197108 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb210014420 con 0x7fb21c0f5410 2024-02-19T22:18:55.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.848+0000 7fb2216e1700 1 -- 172.21.15.184:0/881197108 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb21c045ad0 con 0x7fb21c0f5410 2024-02-19T22:18:55.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.848+0000 7fb2037fe700 1 -- 172.21.15.184:0/881197108 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb21001d980 con 0x7fb21c0f5410 2024-02-19T22:18:55.852 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:55.852 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:55.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.850+0000 7fb2216e1700 1 -- 172.21.15.184:0/881197108 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb20402ff10 msgr2=0x7fb2040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:55.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.850+0000 7fb2216e1700 1 --2- 172.21.15.184:0/881197108 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb20402ff10 0x7fb2040323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fb20c0096a0 tx=0x7fb20c006b40).stop 2024-02-19T22:18:55.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.850+0000 7fb2216e1700 1 -- 172.21.15.184:0/881197108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb21c0f5410 msgr2=0x7fb21c109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:55.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.850+0000 7fb2216e1700 1 --2- 172.21.15.184:0/881197108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb21c0f5410 0x7fb21c109500 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fb210005010 tx=0x7fb21000b040).stop 2024-02-19T22:18:55.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.850+0000 7fb2216e1700 1 -- 172.21.15.184:0/881197108 shutdown_connections 2024-02-19T22:18:55.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.850+0000 7fb2216e1700 1 --2- 172.21.15.184:0/881197108 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb20402ff10 0x7fb2040323d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:55.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.850+0000 7fb2216e1700 1 --2- 172.21.15.184:0/881197108 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb21c0f5410 0x7fb21c109500 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:55.853 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.850+0000 7fb2216e1700 1 -- 172.21.15.184:0/881197108 >> 172.21.15.184:0/881197108 conn(0x7fb21c0f0b70 msgr2=0x7fb21c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:55.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.850+0000 7fb2216e1700 1 -- 172.21.15.184:0/881197108 shutdown_connections 2024-02-19T22:18:55.854 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:55.850+0000 7fb2216e1700 1 -- 172.21.15.184:0/881197108 wait complete. 2024-02-19T22:18:55.854 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:18:56.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:55 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:54.396104+0000 mgr.smithi176.faoogw (mgr.14162) 320 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:57.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:56 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:55.849724+0000 mon.smithi176 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.184:0/881197108' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:57.630 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:57.630 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:56.396457+0000 mgr.smithi176.faoogw (mgr.14162) 321 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:59.435 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.431+0000 7f6e3af27700 1 -- 172.21.15.184:0/783837211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340f5410 msgr2=0x7f6e340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:59.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.431+0000 7f6e3af27700 1 --2- 172.21.15.184:0/783837211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340f5410 0x7f6e340f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f6e28009fb0 tx=0x7f6e28009b30).stop 2024-02-19T22:18:59.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.431+0000 7f6e3af27700 1 -- 172.21.15.184:0/783837211 shutdown_connections 2024-02-19T22:18:59.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.431+0000 7f6e3af27700 1 --2- 172.21.15.184:0/783837211 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340f5410 0x7f6e340f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:59.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.431+0000 7f6e3af27700 1 -- 172.21.15.184:0/783837211 >> 172.21.15.184:0/783837211 conn(0x7f6e340f0b70 msgr2=0x7f6e340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:59.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.431+0000 7f6e3af27700 1 -- 172.21.15.184:0/783837211 shutdown_connections 2024-02-19T22:18:59.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.431+0000 7f6e3af27700 1 -- 172.21.15.184:0/783837211 wait complete. 2024-02-19T22:18:59.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.432+0000 7f6e3af27700 1 Processor -- start 2024-02-19T22:18:59.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.432+0000 7f6e3af27700 1 -- start start 2024-02-19T22:18:59.436 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.432+0000 7f6e3af27700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340f5410 0x7f6e341067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:59.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.432+0000 7f6e3af27700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e34106cc0 con 0x7f6e340f5410 2024-02-19T22:18:59.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.432+0000 7f6e38cc3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340f5410 0x7f6e341067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:59.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.432+0000 7f6e38cc3700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340f5410 0x7f6e341067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:35430/0 (socket says 172.21.15.184:35430) 2024-02-19T22:18:59.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.432+0000 7f6e38cc3700 1 -- 172.21.15.184:0/1691437004 learned_addr learned my addr 172.21.15.184:0/1691437004 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:18:59.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.433+0000 7f6e38cc3700 1 -- 172.21.15.184:0/1691437004 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e28005040 con 0x7f6e340f5410 2024-02-19T22:18:59.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.433+0000 7f6e38cc3700 1 --2- 172.21.15.184:0/1691437004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340f5410 0x7f6e341067d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f6e28005010 tx=0x7f6e2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:59.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.433+0000 7f6e317fa700 1 -- 172.21.15.184:0/1691437004 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e28009320 con 0x7f6e340f5410 2024-02-19T22:18:59.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.433+0000 7f6e3af27700 1 -- 172.21.15.184:0/1691437004 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e34106ec0 con 0x7f6e340f5410 2024-02-19T22:18:59.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.433+0000 7f6e317fa700 1 -- 172.21.15.184:0/1691437004 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e28013380 con 0x7f6e340f5410 2024-02-19T22:18:59.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.433+0000 7f6e317fa700 1 -- 172.21.15.184:0/1691437004 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e28014e50 con 0x7f6e340f5410 2024-02-19T22:18:59.437 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.433+0000 7f6e3af27700 1 -- 172.21.15.184:0/1691437004 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e34107350 con 0x7f6e340f5410 2024-02-19T22:18:59.438 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.434+0000 7f6e317fa700 1 -- 172.21.15.184:0/1691437004 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e280095f0 con 0x7f6e340f5410 2024-02-19T22:18:59.438 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.434+0000 7f6e3af27700 1 -- 172.21.15.184:0/1691437004 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e34045ad0 con 0x7f6e340f5410 2024-02-19T22:18:59.439 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.434+0000 7f6e317fa700 1 --2- 172.21.15.184:0/1691437004 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f6e2402ff10 0x7f6e240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:59.439 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.434+0000 7f6e317fa700 1 -- 172.21.15.184:0/1691437004 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e28040e70 con 0x7f6e340f5410 2024-02-19T22:18:59.439 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.434+0000 7f6e33fff700 1 --2- 172.21.15.184:0/1691437004 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f6e2402ff10 0x7f6e240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:59.439 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.435+0000 7f6e33fff700 1 --2- 172.21.15.184:0/1691437004 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f6e2402ff10 0x7f6e240323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6e200096a0 tx=0x7f6e20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:59.440 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.438+0000 7f6e317fa700 1 -- 172.21.15.184:0/1691437004 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e28014420 con 0x7f6e340f5410 2024-02-19T22:18:59.647 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.645+0000 7f6e3af27700 1 -- 172.21.15.184:0/1691437004 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e341081a0 con 0x7f6e340f5410 2024-02-19T22:18:59.648 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.646+0000 7f6e317fa700 1 -- 172.21.15.184:0/1691437004 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6e28010230 con 0x7f6e340f5410 2024-02-19T22:18:59.648 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:18:59.648 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:59.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.647+0000 7f6e3af27700 1 -- 172.21.15.184:0/1691437004 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f6e2402ff10 msgr2=0x7f6e240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:59.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.647+0000 7f6e3af27700 1 --2- 172.21.15.184:0/1691437004 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f6e2402ff10 0x7f6e240323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6e200096a0 tx=0x7f6e20006b40).stop 2024-02-19T22:18:59.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.647+0000 7f6e3af27700 1 -- 172.21.15.184:0/1691437004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340f5410 msgr2=0x7f6e341067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:59.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.647+0000 7f6e3af27700 1 --2- 172.21.15.184:0/1691437004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340f5410 0x7f6e341067d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f6e28005010 tx=0x7f6e2800b040).stop 2024-02-19T22:18:59.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.647+0000 7f6e3af27700 1 -- 172.21.15.184:0/1691437004 shutdown_connections 2024-02-19T22:18:59.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.647+0000 7f6e3af27700 1 --2- 172.21.15.184:0/1691437004 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f6e2402ff10 0x7f6e240323d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:59.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.647+0000 7f6e3af27700 1 --2- 172.21.15.184:0/1691437004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6e340f5410 0x7f6e341067d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:59.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.647+0000 7f6e3af27700 1 -- 172.21.15.184:0/1691437004 >> 172.21.15.184:0/1691437004 conn(0x7f6e340f0b70 msgr2=0x7f6e3405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:59.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.648+0000 7f6e3af27700 1 -- 172.21.15.184:0/1691437004 shutdown_connections 2024-02-19T22:18:59.650 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:18:59.648+0000 7f6e3af27700 1 -- 172.21.15.184:0/1691437004 wait complete. 2024-02-19T22:18:59.651 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:00.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:59 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:18:58.396835+0000 mgr.smithi176.faoogw (mgr.14162) 322 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:00.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:18:59 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:18:59.647170+0000 mon.smithi176 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.184:0/1691437004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:01.247 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:01.248 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:02.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:00.397177+0000 mgr.smithi176.faoogw (mgr.14162) 323 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:03.320 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.298+0000 7f73c0b72700 1 -- 172.21.15.184:0/1542637264 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73bc0f51a0 msgr2=0x7f73bc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:03.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.298+0000 7f73c0b72700 1 --2- 172.21.15.184:0/1542637264 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73bc0f51a0 0x7f73bc0f55c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f73ac009fb0 tx=0x7f73ac009b30).stop 2024-02-19T22:19:03.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.298+0000 7f73c0b72700 1 -- 172.21.15.184:0/1542637264 shutdown_connections 2024-02-19T22:19:03.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.298+0000 7f73c0b72700 1 --2- 172.21.15.184:0/1542637264 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73bc0f51a0 0x7f73bc0f55c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:03.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.298+0000 7f73c0b72700 1 -- 172.21.15.184:0/1542637264 >> 172.21.15.184:0/1542637264 conn(0x7f73bc0f09a0 msgr2=0x7f73bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:03.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.298+0000 7f73c0b72700 1 -- 172.21.15.184:0/1542637264 shutdown_connections 2024-02-19T22:19:03.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.298+0000 7f73c0b72700 1 -- 172.21.15.184:0/1542637264 wait complete. 2024-02-19T22:19:03.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.298+0000 7f73c0b72700 1 Processor -- start 2024-02-19T22:19:03.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.299+0000 7f73c0b72700 1 -- start start 2024-02-19T22:19:03.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.299+0000 7f73c0b72700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73bc0f51a0 0x7f73bc1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:03.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.299+0000 7f73c0b72700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73bc1059c0 con 0x7f73bc0f51a0 2024-02-19T22:19:03.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.299+0000 7f73bb7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73bc0f51a0 0x7f73bc1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:03.321 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.299+0000 7f73bb7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73bc0f51a0 0x7f73bc1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:35438/0 (socket says 172.21.15.184:35438) 2024-02-19T22:19:03.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.299+0000 7f73bb7fe700 1 -- 172.21.15.184:0/1169032289 learned_addr learned my addr 172.21.15.184:0/1169032289 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:03.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.299+0000 7f73bb7fe700 1 -- 172.21.15.184:0/1169032289 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac005040 con 0x7f73bc0f51a0 2024-02-19T22:19:03.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.299+0000 7f73bb7fe700 1 --2- 172.21.15.184:0/1169032289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73bc0f51a0 0x7f73bc1054d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f73ac0089f0 tx=0x7f73ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:03.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.299+0000 7f73a3fff700 1 -- 172.21.15.184:0/1169032289 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73ac009320 con 0x7f73bc0f51a0 2024-02-19T22:19:03.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.300+0000 7f73c0b72700 1 -- 172.21.15.184:0/1169032289 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73bc105bc0 con 0x7f73bc0f51a0 2024-02-19T22:19:03.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.300+0000 7f73a3fff700 1 -- 172.21.15.184:0/1169032289 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f73ac013380 con 0x7f73bc0f51a0 2024-02-19T22:19:03.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.300+0000 7f73a3fff700 1 -- 172.21.15.184:0/1169032289 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73ac014e20 con 0x7f73bc0f51a0 2024-02-19T22:19:03.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.300+0000 7f73c0b72700 1 -- 172.21.15.184:0/1169032289 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73bc106050 con 0x7f73bc0f51a0 2024-02-19T22:19:03.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.300+0000 7f73a3fff700 1 -- 172.21.15.184:0/1169032289 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f73ac0095f0 con 0x7f73bc0f51a0 2024-02-19T22:19:03.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.301+0000 7f73c0b72700 1 -- 172.21.15.184:0/1169032289 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73bc045ad0 con 0x7f73bc0f51a0 2024-02-19T22:19:03.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.301+0000 7f73a3fff700 1 --2- 172.21.15.184:0/1169032289 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f73a402ff10 0x7f73a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:03.322 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.301+0000 7f73a3fff700 1 -- 172.21.15.184:0/1169032289 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f73ac040e90 con 0x7f73bc0f51a0 2024-02-19T22:19:03.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.301+0000 7f73baffd700 1 --2- 172.21.15.184:0/1169032289 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f73a402ff10 0x7f73a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:03.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.301+0000 7f73baffd700 1 --2- 172.21.15.184:0/1169032289 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f73a402ff10 0x7f73a40323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f73b00096a0 tx=0x7f73b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:03.323 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.305+0000 7f73a3fff700 1 -- 172.21.15.184:0/1169032289 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f73ac014420 con 0x7f73bc0f51a0 2024-02-19T22:19:03.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.519+0000 7f73c0b72700 1 -- 172.21.15.184:0/1169032289 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73bc106f00 con 0x7f73bc0f51a0 2024-02-19T22:19:03.522 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.520+0000 7f73a3fff700 1 -- 172.21.15.184:0/1169032289 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f73ac01d980 con 0x7f73bc0f51a0 2024-02-19T22:19:03.522 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:03.523 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:03.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.522+0000 7f73c0b72700 1 -- 172.21.15.184:0/1169032289 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f73a402ff10 msgr2=0x7f73a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:03.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.522+0000 7f73c0b72700 1 --2- 172.21.15.184:0/1169032289 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f73a402ff10 0x7f73a40323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f73b00096a0 tx=0x7f73b0006b40).stop 2024-02-19T22:19:03.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.522+0000 7f73c0b72700 1 -- 172.21.15.184:0/1169032289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73bc0f51a0 msgr2=0x7f73bc1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:03.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.522+0000 7f73c0b72700 1 --2- 172.21.15.184:0/1169032289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73bc0f51a0 0x7f73bc1054d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f73ac0089f0 tx=0x7f73ac00b040).stop 2024-02-19T22:19:03.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.522+0000 7f73c0b72700 1 -- 172.21.15.184:0/1169032289 shutdown_connections 2024-02-19T22:19:03.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.522+0000 7f73c0b72700 1 --2- 172.21.15.184:0/1169032289 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f73a402ff10 0x7f73a40323d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:03.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.522+0000 7f73c0b72700 1 --2- 172.21.15.184:0/1169032289 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f73bc0f51a0 0x7f73bc1054d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:03.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.522+0000 7f73c0b72700 1 -- 172.21.15.184:0/1169032289 >> 172.21.15.184:0/1169032289 conn(0x7f73bc0f09a0 msgr2=0x7f73bc0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:03.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.522+0000 7f73c0b72700 1 -- 172.21.15.184:0/1169032289 shutdown_connections 2024-02-19T22:19:03.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:03.522+0000 7f73c0b72700 1 -- 172.21.15.184:0/1169032289 wait complete. 2024-02-19T22:19:03.525 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:04.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:02.397598+0000 mgr.smithi176.faoogw (mgr.14162) 324 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:04.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:03 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:03.521595+0000 mon.smithi176 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.184:0/1169032289' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:05.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:05.279 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:06.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:04.397950+0000 mgr.smithi176.faoogw (mgr.14162) 325 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:07.165 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.161+0000 7fdd9045d700 1 -- 172.21.15.184:0/560122811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880f6480 msgr2=0x7fdd880f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:07.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.161+0000 7fdd9045d700 1 --2- 172.21.15.184:0/560122811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880f6480 0x7fdd880f68a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fdd78009fb0 tx=0x7fdd78009b30).stop 2024-02-19T22:19:07.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.162+0000 7fdd9045d700 1 -- 172.21.15.184:0/560122811 shutdown_connections 2024-02-19T22:19:07.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.162+0000 7fdd9045d700 1 --2- 172.21.15.184:0/560122811 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880f6480 0x7fdd880f68a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:07.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.162+0000 7fdd9045d700 1 -- 172.21.15.184:0/560122811 >> 172.21.15.184:0/560122811 conn(0x7fdd880f1c00 msgr2=0x7fdd880f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:07.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.162+0000 7fdd9045d700 1 -- 172.21.15.184:0/560122811 shutdown_connections 2024-02-19T22:19:07.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.162+0000 7fdd9045d700 1 -- 172.21.15.184:0/560122811 wait complete. 2024-02-19T22:19:07.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.162+0000 7fdd9045d700 1 Processor -- start 2024-02-19T22:19:07.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.162+0000 7fdd9045d700 1 -- start start 2024-02-19T22:19:07.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.163+0000 7fdd9045d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880f6480 0x7fdd881067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:07.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.163+0000 7fdd9045d700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd88106ce0 con 0x7fdd880f6480 2024-02-19T22:19:07.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.163+0000 7fdd8e1f9700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880f6480 0x7fdd881067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:07.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.163+0000 7fdd8e1f9700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880f6480 0x7fdd881067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:44308/0 (socket says 172.21.15.184:44308) 2024-02-19T22:19:07.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.163+0000 7fdd8e1f9700 1 -- 172.21.15.184:0/684698068 learned_addr learned my addr 172.21.15.184:0/684698068 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:07.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.163+0000 7fdd8e1f9700 1 -- 172.21.15.184:0/684698068 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd78005040 con 0x7fdd880f6480 2024-02-19T22:19:07.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.163+0000 7fdd8e1f9700 1 --2- 172.21.15.184:0/684698068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880f6480 0x7fdd881067f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fdd78005010 tx=0x7fdd7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:07.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.163+0000 7fdd7effd700 1 -- 172.21.15.184:0/684698068 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd78009320 con 0x7fdd880f6480 2024-02-19T22:19:07.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.163+0000 7fdd7effd700 1 -- 172.21.15.184:0/684698068 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd78013380 con 0x7fdd880f6480 2024-02-19T22:19:07.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.163+0000 7fdd9045d700 1 -- 172.21.15.184:0/684698068 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd88106ee0 con 0x7fdd880f6480 2024-02-19T22:19:07.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.163+0000 7fdd7effd700 1 -- 172.21.15.184:0/684698068 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd78014e50 con 0x7fdd880f6480 2024-02-19T22:19:07.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.164+0000 7fdd9045d700 1 -- 172.21.15.184:0/684698068 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd88107370 con 0x7fdd880f6480 2024-02-19T22:19:07.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.164+0000 7fdd7effd700 1 -- 172.21.15.184:0/684698068 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd78014420 con 0x7fdd880f6480 2024-02-19T22:19:07.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.164+0000 7fdd9045d700 1 -- 172.21.15.184:0/684698068 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd88045ad0 con 0x7fdd880f6480 2024-02-19T22:19:07.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.165+0000 7fdd7effd700 1 --2- 172.21.15.184:0/684698068 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fdd7402ff10 0x7fdd740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:07.168 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.165+0000 7fdd7effd700 1 -- 172.21.15.184:0/684698068 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd78040e30 con 0x7fdd880f6480 2024-02-19T22:19:07.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.165+0000 7fdd8d9f8700 1 --2- 172.21.15.184:0/684698068 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fdd7402ff10 0x7fdd740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:07.169 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.165+0000 7fdd8d9f8700 1 --2- 172.21.15.184:0/684698068 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fdd7402ff10 0x7fdd740323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fdd840096a0 tx=0x7fdd84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:07.171 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.169+0000 7fdd7effd700 1 -- 172.21.15.184:0/684698068 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd780095f0 con 0x7fdd880f6480 2024-02-19T22:19:07.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.374+0000 7fdd9045d700 1 -- 172.21.15.184:0/684698068 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd880fc090 con 0x7fdd880f6480 2024-02-19T22:19:07.376 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.374+0000 7fdd7effd700 1 -- 172.21.15.184:0/684698068 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdd7801db70 con 0x7fdd880f6480 2024-02-19T22:19:07.377 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:07.377 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:07.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.376+0000 7fdd9045d700 1 -- 172.21.15.184:0/684698068 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fdd7402ff10 msgr2=0x7fdd740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:07.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.376+0000 7fdd9045d700 1 --2- 172.21.15.184:0/684698068 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fdd7402ff10 0x7fdd740323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fdd840096a0 tx=0x7fdd84006b40).stop 2024-02-19T22:19:07.378 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.376+0000 7fdd9045d700 1 -- 172.21.15.184:0/684698068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880f6480 msgr2=0x7fdd881067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:07.379 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.376+0000 7fdd9045d700 1 --2- 172.21.15.184:0/684698068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880f6480 0x7fdd881067f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fdd78005010 tx=0x7fdd7800b040).stop 2024-02-19T22:19:07.379 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.376+0000 7fdd9045d700 1 -- 172.21.15.184:0/684698068 shutdown_connections 2024-02-19T22:19:07.379 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.376+0000 7fdd9045d700 1 --2- 172.21.15.184:0/684698068 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fdd7402ff10 0x7fdd740323d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:07.379 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.376+0000 7fdd9045d700 1 --2- 172.21.15.184:0/684698068 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd880f6480 0x7fdd881067f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:07.379 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.376+0000 7fdd9045d700 1 -- 172.21.15.184:0/684698068 >> 172.21.15.184:0/684698068 conn(0x7fdd880f1c00 msgr2=0x7fdd880f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:07.379 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.376+0000 7fdd9045d700 1 -- 172.21.15.184:0/684698068 shutdown_connections 2024-02-19T22:19:07.379 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:07.376+0000 7fdd9045d700 1 -- 172.21.15.184:0/684698068 wait complete. 2024-02-19T22:19:07.379 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:08.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:06.398298+0000 mgr.smithi176.faoogw (mgr.14162) 326 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:08.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:07.375960+0000 mon.smithi176 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.184:0/684698068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:09.074 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:09.074 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:10.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:08.398665+0000 mgr.smithi176.faoogw (mgr.14162) 327 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:10.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.950+0000 7f25526ac700 1 -- 172.21.15.184:0/4165598259 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f254c0f6950 msgr2=0x7f254c0f6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:10.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.950+0000 7f25526ac700 1 --2- 172.21.15.184:0/4165598259 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f254c0f6950 0x7f254c0f6d70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f2540009fb0 tx=0x7f2540009b30).stop 2024-02-19T22:19:10.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.950+0000 7f25526ac700 1 -- 172.21.15.184:0/4165598259 shutdown_connections 2024-02-19T22:19:10.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.950+0000 7f25526ac700 1 --2- 172.21.15.184:0/4165598259 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f254c0f6950 0x7f254c0f6d70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:10.954 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.950+0000 7f25526ac700 1 -- 172.21.15.184:0/4165598259 >> 172.21.15.184:0/4165598259 conn(0x7f254c0f0260 msgr2=0x7f254c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:10.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.950+0000 7f25526ac700 1 -- 172.21.15.184:0/4165598259 shutdown_connections 2024-02-19T22:19:10.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.950+0000 7f25526ac700 1 -- 172.21.15.184:0/4165598259 wait complete. 2024-02-19T22:19:10.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.951+0000 7f25526ac700 1 Processor -- start 2024-02-19T22:19:10.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.951+0000 7f25526ac700 1 -- start start 2024-02-19T22:19:10.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.951+0000 7f25526ac700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f254c0f6950 0x7f254c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:10.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.951+0000 7f25526ac700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c106cc0 con 0x7f254c0f6950 2024-02-19T22:19:10.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.951+0000 7f254bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f254c0f6950 0x7f254c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:10.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.951+0000 7f254bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f254c0f6950 0x7f254c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:44334/0 (socket says 172.21.15.184:44334) 2024-02-19T22:19:10.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.951+0000 7f254bfff700 1 -- 172.21.15.184:0/1297817111 learned_addr learned my addr 172.21.15.184:0/1297817111 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:10.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.952+0000 7f254bfff700 1 -- 172.21.15.184:0/1297817111 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2540005040 con 0x7f254c0f6950 2024-02-19T22:19:10.955 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.952+0000 7f254bfff700 1 --2- 172.21.15.184:0/1297817111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f254c0f6950 0x7f254c1067d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f2540005010 tx=0x7f254000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:10.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.952+0000 7f2548ff9700 1 -- 172.21.15.184:0/1297817111 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2540009320 con 0x7f254c0f6950 2024-02-19T22:19:10.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.952+0000 7f25526ac700 1 -- 172.21.15.184:0/1297817111 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254c106ec0 con 0x7f254c0f6950 2024-02-19T22:19:10.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.952+0000 7f2548ff9700 1 -- 172.21.15.184:0/1297817111 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2540013380 con 0x7f254c0f6950 2024-02-19T22:19:10.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.952+0000 7f2548ff9700 1 -- 172.21.15.184:0/1297817111 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2540014e50 con 0x7f254c0f6950 2024-02-19T22:19:10.956 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.952+0000 7f25526ac700 1 -- 172.21.15.184:0/1297817111 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f254c107350 con 0x7f254c0f6950 2024-02-19T22:19:10.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.953+0000 7f2548ff9700 1 -- 172.21.15.184:0/1297817111 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25400263a0 con 0x7f254c0f6950 2024-02-19T22:19:10.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.953+0000 7f25526ac700 1 -- 172.21.15.184:0/1297817111 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f254c045ad0 con 0x7f254c0f6950 2024-02-19T22:19:10.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.953+0000 7f2548ff9700 1 --2- 172.21.15.184:0/1297817111 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f253402ff10 0x7f25340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:10.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.953+0000 7f2548ff9700 1 -- 172.21.15.184:0/1297817111 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2540041280 con 0x7f254c0f6950 2024-02-19T22:19:10.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.953+0000 7f254b7fe700 1 --2- 172.21.15.184:0/1297817111 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f253402ff10 0x7f25340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:10.957 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.954+0000 7f254b7fe700 1 --2- 172.21.15.184:0/1297817111 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f253402ff10 0x7f25340323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f253c0096a0 tx=0x7f253c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:10.960 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:10.958+0000 7f2548ff9700 1 -- 172.21.15.184:0/1297817111 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25400126f0 con 0x7f254c0f6950 2024-02-19T22:19:11.163 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:11.161+0000 7f25526ac700 1 -- 172.21.15.184:0/1297817111 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f254c108200 con 0x7f254c0f6950 2024-02-19T22:19:11.164 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:11.162+0000 7f2548ff9700 1 -- 172.21.15.184:0/1297817111 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2540012330 con 0x7f254c0f6950 2024-02-19T22:19:11.164 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:11.164 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:11.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:11.164+0000 7f25526ac700 1 -- 172.21.15.184:0/1297817111 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f253402ff10 msgr2=0x7f25340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:11.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:11.164+0000 7f25526ac700 1 --2- 172.21.15.184:0/1297817111 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f253402ff10 0x7f25340323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f253c0096a0 tx=0x7f253c006b40).stop 2024-02-19T22:19:11.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:11.164+0000 7f25526ac700 1 -- 172.21.15.184:0/1297817111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f254c0f6950 msgr2=0x7f254c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:11.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:11.164+0000 7f25526ac700 1 --2- 172.21.15.184:0/1297817111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f254c0f6950 0x7f254c1067d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f2540005010 tx=0x7f254000b040).stop 2024-02-19T22:19:11.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:11.164+0000 7f25526ac700 1 -- 172.21.15.184:0/1297817111 shutdown_connections 2024-02-19T22:19:11.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:11.164+0000 7f25526ac700 1 --2- 172.21.15.184:0/1297817111 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f253402ff10 0x7f25340323d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:11.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:11.164+0000 7f25526ac700 1 --2- 172.21.15.184:0/1297817111 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f254c0f6950 0x7f254c1067d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:11.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:11.164+0000 7f25526ac700 1 -- 172.21.15.184:0/1297817111 >> 172.21.15.184:0/1297817111 conn(0x7f254c0f0260 msgr2=0x7f254c0f4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:11.166 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:11.164+0000 7f25526ac700 1 -- 172.21.15.184:0/1297817111 shutdown_connections 2024-02-19T22:19:11.167 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:11.164+0000 7f25526ac700 1 -- 172.21.15.184:0/1297817111 wait complete. 2024-02-19T22:19:11.167 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:12.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:10.399034+0000 mgr.smithi176.faoogw (mgr.14162) 328 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:12.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:11.163321+0000 mon.smithi176 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.184:0/1297817111' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:12.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:12.820 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:14.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:13 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:12.399417+0000 mgr.smithi176.faoogw (mgr.14162) 329 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:14.728 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.726+0000 7fac0759e700 1 -- 172.21.15.184:0/1681542198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac080f51a0 msgr2=0x7fac080f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:14.728 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.726+0000 7fac0759e700 1 --2- 172.21.15.184:0/1681542198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac080f51a0 0x7fac080f55c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fabfc009fb0 tx=0x7fabfc009b30).stop 2024-02-19T22:19:14.728 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.726+0000 7fac0759e700 1 -- 172.21.15.184:0/1681542198 shutdown_connections 2024-02-19T22:19:14.729 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.726+0000 7fac0759e700 1 --2- 172.21.15.184:0/1681542198 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac080f51a0 0x7fac080f55c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:14.729 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.726+0000 7fac0759e700 1 -- 172.21.15.184:0/1681542198 >> 172.21.15.184:0/1681542198 conn(0x7fac080f09a0 msgr2=0x7fac080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:14.729 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.726+0000 7fac0759e700 1 -- 172.21.15.184:0/1681542198 shutdown_connections 2024-02-19T22:19:14.729 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.726+0000 7fac0759e700 1 -- 172.21.15.184:0/1681542198 wait complete. 2024-02-19T22:19:14.729 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.726+0000 7fac0759e700 1 Processor -- start 2024-02-19T22:19:14.729 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.726+0000 7fac0759e700 1 -- start start 2024-02-19T22:19:14.729 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.726+0000 7fac0759e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac080f51a0 0x7fac08109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:14.729 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.726+0000 7fac0759e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac08060280 con 0x7fac080f51a0 2024-02-19T22:19:14.729 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.727+0000 7fac0659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac080f51a0 0x7fac08109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:14.729 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.727+0000 7fac0659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac080f51a0 0x7fac08109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:44346/0 (socket says 172.21.15.184:44346) 2024-02-19T22:19:14.729 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.727+0000 7fac0659c700 1 -- 172.21.15.184:0/3802554514 learned_addr learned my addr 172.21.15.184:0/3802554514 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:14.729 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.727+0000 7fac0659c700 1 -- 172.21.15.184:0/3802554514 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabfc005040 con 0x7fac080f51a0 2024-02-19T22:19:14.729 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.727+0000 7fac0659c700 1 --2- 172.21.15.184:0/3802554514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac080f51a0 0x7fac08109290 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fabfc0089f0 tx=0x7fabfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:14.729 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.727+0000 7fabf6ffd700 1 -- 172.21.15.184:0/3802554514 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabfc009320 con 0x7fac080f51a0 2024-02-19T22:19:14.730 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.727+0000 7fabf6ffd700 1 -- 172.21.15.184:0/3802554514 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fabfc013380 con 0x7fac080f51a0 2024-02-19T22:19:14.730 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.727+0000 7fac0759e700 1 -- 172.21.15.184:0/3802554514 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac08060480 con 0x7fac080f51a0 2024-02-19T22:19:14.730 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.727+0000 7fabf6ffd700 1 -- 172.21.15.184:0/3802554514 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabfc014e20 con 0x7fac080f51a0 2024-02-19T22:19:14.730 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.727+0000 7fac0759e700 1 -- 172.21.15.184:0/3802554514 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac081099c0 con 0x7fac080f51a0 2024-02-19T22:19:14.730 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.728+0000 7fabf6ffd700 1 -- 172.21.15.184:0/3802554514 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fabfc0263a0 con 0x7fac080f51a0 2024-02-19T22:19:14.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.728+0000 7fabf6ffd700 1 --2- 172.21.15.184:0/3802554514 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fabf002ff10 0x7fabf00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:14.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.728+0000 7fabf6ffd700 1 -- 172.21.15.184:0/3802554514 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fabfc040f90 con 0x7fac080f51a0 2024-02-19T22:19:14.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.728+0000 7fac0759e700 1 -- 172.21.15.184:0/3802554514 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac08045ad0 con 0x7fac080f51a0 2024-02-19T22:19:14.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.728+0000 7fac05d9b700 1 --2- 172.21.15.184:0/3802554514 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fabf002ff10 0x7fabf00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:14.731 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.729+0000 7fac05d9b700 1 --2- 172.21.15.184:0/3802554514 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fabf002ff10 0x7fabf00323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fabf80096a0 tx=0x7fabf8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:14.734 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.733+0000 7fabf6ffd700 1 -- 172.21.15.184:0/3802554514 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fabfc012920 con 0x7fac080f51a0 2024-02-19T22:19:14.938 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.937+0000 7fac0759e700 1 -- 172.21.15.184:0/3802554514 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac080fad30 con 0x7fac080f51a0 2024-02-19T22:19:14.939 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.938+0000 7fabf6ffd700 1 -- 172.21.15.184:0/3802554514 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fabfc0126f0 con 0x7fac080f51a0 2024-02-19T22:19:14.939 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:14.939 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:14.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.939+0000 7fac0759e700 1 -- 172.21.15.184:0/3802554514 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fabf002ff10 msgr2=0x7fabf00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:14.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.939+0000 7fac0759e700 1 --2- 172.21.15.184:0/3802554514 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fabf002ff10 0x7fabf00323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fabf80096a0 tx=0x7fabf8006b40).stop 2024-02-19T22:19:14.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.939+0000 7fac0759e700 1 -- 172.21.15.184:0/3802554514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac080f51a0 msgr2=0x7fac08109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:14.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.939+0000 7fac0759e700 1 --2- 172.21.15.184:0/3802554514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac080f51a0 0x7fac08109290 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fabfc0089f0 tx=0x7fabfc00b040).stop 2024-02-19T22:19:14.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.939+0000 7fac0759e700 1 -- 172.21.15.184:0/3802554514 shutdown_connections 2024-02-19T22:19:14.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.939+0000 7fac0759e700 1 --2- 172.21.15.184:0/3802554514 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fabf002ff10 0x7fabf00323d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:14.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.939+0000 7fac0759e700 1 --2- 172.21.15.184:0/3802554514 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac080f51a0 0x7fac08109290 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:14.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.939+0000 7fac0759e700 1 -- 172.21.15.184:0/3802554514 >> 172.21.15.184:0/3802554514 conn(0x7fac080f09a0 msgr2=0x7fac080f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:14.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.939+0000 7fac0759e700 1 -- 172.21.15.184:0/3802554514 shutdown_connections 2024-02-19T22:19:14.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:14.939+0000 7fac0759e700 1 -- 172.21.15.184:0/3802554514 wait complete. 2024-02-19T22:19:14.941 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:16.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:14.399818+0000 mgr.smithi176.faoogw (mgr.14162) 330 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:16.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:14.938073+0000 mon.smithi176 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.184:0/3802554514' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:16.730 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:16.731 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:18.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:17 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:16.400397+0000 mgr.smithi176.faoogw (mgr.14162) 331 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:18.658 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.651+0000 7f4b641ab700 1 -- 172.21.15.184:0/3701051547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c0f6510 msgr2=0x7f4b5c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:18.658 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.651+0000 7f4b641ab700 1 --2- 172.21.15.184:0/3701051547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c0f6510 0x7f4b5c0f6930 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f4b4c009fb0 tx=0x7f4b4c009b30).stop 2024-02-19T22:19:18.658 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.651+0000 7f4b641ab700 1 -- 172.21.15.184:0/3701051547 shutdown_connections 2024-02-19T22:19:18.658 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.651+0000 7f4b641ab700 1 --2- 172.21.15.184:0/3701051547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c0f6510 0x7f4b5c0f6930 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:18.658 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.651+0000 7f4b641ab700 1 -- 172.21.15.184:0/3701051547 >> 172.21.15.184:0/3701051547 conn(0x7f4b5c0f1c70 msgr2=0x7f4b5c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:18.659 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.651+0000 7f4b641ab700 1 -- 172.21.15.184:0/3701051547 shutdown_connections 2024-02-19T22:19:18.659 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.651+0000 7f4b641ab700 1 -- 172.21.15.184:0/3701051547 wait complete. 2024-02-19T22:19:18.659 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.652+0000 7f4b641ab700 1 Processor -- start 2024-02-19T22:19:18.659 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.652+0000 7f4b641ab700 1 -- start start 2024-02-19T22:19:18.659 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.652+0000 7f4b641ab700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c0f6510 0x7f4b5c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:18.660 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.652+0000 7f4b641ab700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c106cd0 con 0x7f4b5c0f6510 2024-02-19T22:19:18.660 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.652+0000 7f4b61f47700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c0f6510 0x7f4b5c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:18.660 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.652+0000 7f4b61f47700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c0f6510 0x7f4b5c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:49370/0 (socket says 172.21.15.184:49370) 2024-02-19T22:19:18.660 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.652+0000 7f4b61f47700 1 -- 172.21.15.184:0/2364711042 learned_addr learned my addr 172.21.15.184:0/2364711042 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:18.660 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.653+0000 7f4b61f47700 1 -- 172.21.15.184:0/2364711042 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4c005040 con 0x7f4b5c0f6510 2024-02-19T22:19:18.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.653+0000 7f4b61f47700 1 --2- 172.21.15.184:0/2364711042 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c0f6510 0x7f4b5c1067e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f4b4c005010 tx=0x7f4b4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:18.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.653+0000 7f4b527fc700 1 -- 172.21.15.184:0/2364711042 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b4c009320 con 0x7f4b5c0f6510 2024-02-19T22:19:18.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.653+0000 7f4b641ab700 1 -- 172.21.15.184:0/2364711042 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5c106ed0 con 0x7f4b5c0f6510 2024-02-19T22:19:18.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.653+0000 7f4b527fc700 1 -- 172.21.15.184:0/2364711042 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b4c013380 con 0x7f4b5c0f6510 2024-02-19T22:19:18.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.653+0000 7f4b527fc700 1 -- 172.21.15.184:0/2364711042 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b4c014e50 con 0x7f4b5c0f6510 2024-02-19T22:19:18.661 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.653+0000 7f4b641ab700 1 -- 172.21.15.184:0/2364711042 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5c107360 con 0x7f4b5c0f6510 2024-02-19T22:19:18.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.654+0000 7f4b641ab700 1 -- 172.21.15.184:0/2364711042 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5c045ad0 con 0x7f4b5c0f6510 2024-02-19T22:19:18.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.654+0000 7f4b527fc700 1 -- 172.21.15.184:0/2364711042 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b4c0095f0 con 0x7f4b5c0f6510 2024-02-19T22:19:18.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.654+0000 7f4b527fc700 1 --2- 172.21.15.184:0/2364711042 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f4b4802ff10 0x7f4b480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:18.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.654+0000 7f4b527fc700 1 -- 172.21.15.184:0/2364711042 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b4c040e70 con 0x7f4b5c0f6510 2024-02-19T22:19:18.662 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.654+0000 7f4b61746700 1 --2- 172.21.15.184:0/2364711042 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f4b4802ff10 0x7f4b480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:18.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.655+0000 7f4b61746700 1 --2- 172.21.15.184:0/2364711042 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f4b4802ff10 0x7f4b480323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4b580096a0 tx=0x7f4b58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:18.663 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.659+0000 7f4b527fc700 1 -- 172.21.15.184:0/2364711042 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b4c014420 con 0x7f4b5c0f6510 2024-02-19T22:19:18.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.866+0000 7f4b641ab700 1 -- 172.21.15.184:0/2364711042 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b5c1081b0 con 0x7f4b5c0f6510 2024-02-19T22:19:18.869 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.866+0000 7f4b527fc700 1 -- 172.21.15.184:0/2364711042 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4b4c010230 con 0x7f4b5c0f6510 2024-02-19T22:19:18.869 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:18.869 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:18.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.868+0000 7f4b641ab700 1 -- 172.21.15.184:0/2364711042 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f4b4802ff10 msgr2=0x7f4b480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:18.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.868+0000 7f4b641ab700 1 --2- 172.21.15.184:0/2364711042 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f4b4802ff10 0x7f4b480323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f4b580096a0 tx=0x7f4b58006b40).stop 2024-02-19T22:19:18.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.868+0000 7f4b641ab700 1 -- 172.21.15.184:0/2364711042 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c0f6510 msgr2=0x7f4b5c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:18.871 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.868+0000 7f4b641ab700 1 --2- 172.21.15.184:0/2364711042 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c0f6510 0x7f4b5c1067e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f4b4c005010 tx=0x7f4b4c00b040).stop 2024-02-19T22:19:18.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.868+0000 7f4b641ab700 1 -- 172.21.15.184:0/2364711042 shutdown_connections 2024-02-19T22:19:18.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.868+0000 7f4b641ab700 1 --2- 172.21.15.184:0/2364711042 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f4b4802ff10 0x7f4b480323d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:18.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.868+0000 7f4b641ab700 1 --2- 172.21.15.184:0/2364711042 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4b5c0f6510 0x7f4b5c1067e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:18.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.868+0000 7f4b641ab700 1 -- 172.21.15.184:0/2364711042 >> 172.21.15.184:0/2364711042 conn(0x7f4b5c0f1c70 msgr2=0x7f4b5c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:18.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.868+0000 7f4b641ab700 1 -- 172.21.15.184:0/2364711042 shutdown_connections 2024-02-19T22:19:18.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:18.868+0000 7f4b641ab700 1 -- 172.21.15.184:0/2364711042 wait complete. 2024-02-19T22:19:18.873 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:20.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:18.400642+0000 mgr.smithi176.faoogw (mgr.14162) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:20.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:18.867735+0000 mon.smithi176 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.184:0/2364711042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:20.457 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:20.457 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:22.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:20.400952+0000 mgr.smithi176.faoogw (mgr.14162) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:22.350 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.333+0000 7fef43fff700 1 -- 172.21.15.184:0/3143898462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef440f51a0 msgr2=0x7fef440f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:22.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.333+0000 7fef43fff700 1 --2- 172.21.15.184:0/3143898462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef440f51a0 0x7fef440f55c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fef34009fb0 tx=0x7fef34009b30).stop 2024-02-19T22:19:22.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.333+0000 7fef43fff700 1 -- 172.21.15.184:0/3143898462 shutdown_connections 2024-02-19T22:19:22.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.333+0000 7fef43fff700 1 --2- 172.21.15.184:0/3143898462 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef440f51a0 0x7fef440f55c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:22.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.333+0000 7fef43fff700 1 -- 172.21.15.184:0/3143898462 >> 172.21.15.184:0/3143898462 conn(0x7fef440f09a0 msgr2=0x7fef440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:22.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.333+0000 7fef43fff700 1 -- 172.21.15.184:0/3143898462 shutdown_connections 2024-02-19T22:19:22.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.333+0000 7fef43fff700 1 -- 172.21.15.184:0/3143898462 wait complete. 2024-02-19T22:19:22.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.333+0000 7fef43fff700 1 Processor -- start 2024-02-19T22:19:22.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.333+0000 7fef43fff700 1 -- start start 2024-02-19T22:19:22.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.334+0000 7fef43fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef440f51a0 0x7fef441054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:22.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.334+0000 7fef43fff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef441059c0 con 0x7fef440f51a0 2024-02-19T22:19:22.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.334+0000 7fef42ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef440f51a0 0x7fef441054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:22.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.334+0000 7fef42ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef440f51a0 0x7fef441054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:49382/0 (socket says 172.21.15.184:49382) 2024-02-19T22:19:22.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.334+0000 7fef42ffd700 1 -- 172.21.15.184:0/1515706769 learned_addr learned my addr 172.21.15.184:0/1515706769 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:22.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.334+0000 7fef42ffd700 1 -- 172.21.15.184:0/1515706769 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef34005040 con 0x7fef440f51a0 2024-02-19T22:19:22.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.334+0000 7fef42ffd700 1 --2- 172.21.15.184:0/1515706769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef440f51a0 0x7fef441054d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fef340089f0 tx=0x7fef3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:22.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.334+0000 7fef2b7fe700 1 -- 172.21.15.184:0/1515706769 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef34009320 con 0x7fef440f51a0 2024-02-19T22:19:22.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.334+0000 7fef43fff700 1 -- 172.21.15.184:0/1515706769 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef44105bc0 con 0x7fef440f51a0 2024-02-19T22:19:22.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.334+0000 7fef2b7fe700 1 -- 172.21.15.184:0/1515706769 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef34013380 con 0x7fef440f51a0 2024-02-19T22:19:22.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.334+0000 7fef2b7fe700 1 -- 172.21.15.184:0/1515706769 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef34014e20 con 0x7fef440f51a0 2024-02-19T22:19:22.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.335+0000 7fef43fff700 1 -- 172.21.15.184:0/1515706769 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef44106050 con 0x7fef440f51a0 2024-02-19T22:19:22.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.335+0000 7fef2b7fe700 1 -- 172.21.15.184:0/1515706769 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef34014420 con 0x7fef440f51a0 2024-02-19T22:19:22.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.335+0000 7fef43fff700 1 -- 172.21.15.184:0/1515706769 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef44045ad0 con 0x7fef440f51a0 2024-02-19T22:19:22.352 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.335+0000 7fef2b7fe700 1 --2- 172.21.15.184:0/1515706769 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fef2c02ff10 0x7fef2c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:22.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.335+0000 7fef2b7fe700 1 -- 172.21.15.184:0/1515706769 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef34040f90 con 0x7fef440f51a0 2024-02-19T22:19:22.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.336+0000 7fef427fc700 1 --2- 172.21.15.184:0/1515706769 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fef2c02ff10 0x7fef2c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:22.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.336+0000 7fef427fc700 1 --2- 172.21.15.184:0/1515706769 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fef2c02ff10 0x7fef2c0323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fef380096a0 tx=0x7fef38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:22.353 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.340+0000 7fef2b7fe700 1 -- 172.21.15.184:0/1515706769 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef3406d2b0 con 0x7fef440f51a0 2024-02-19T22:19:22.551 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.549+0000 7fef43fff700 1 -- 172.21.15.184:0/1515706769 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef44106f00 con 0x7fef440f51a0 2024-02-19T22:19:22.552 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.550+0000 7fef2b7fe700 1 -- 172.21.15.184:0/1515706769 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fef340095f0 con 0x7fef440f51a0 2024-02-19T22:19:22.552 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:22.552 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:22.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.551+0000 7fef43fff700 1 -- 172.21.15.184:0/1515706769 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fef2c02ff10 msgr2=0x7fef2c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:22.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.551+0000 7fef43fff700 1 --2- 172.21.15.184:0/1515706769 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fef2c02ff10 0x7fef2c0323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fef380096a0 tx=0x7fef38006b40).stop 2024-02-19T22:19:22.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.551+0000 7fef43fff700 1 -- 172.21.15.184:0/1515706769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef440f51a0 msgr2=0x7fef441054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:22.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.551+0000 7fef43fff700 1 --2- 172.21.15.184:0/1515706769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef440f51a0 0x7fef441054d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fef340089f0 tx=0x7fef3400b040).stop 2024-02-19T22:19:22.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.551+0000 7fef43fff700 1 -- 172.21.15.184:0/1515706769 shutdown_connections 2024-02-19T22:19:22.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.551+0000 7fef43fff700 1 --2- 172.21.15.184:0/1515706769 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fef2c02ff10 0x7fef2c0323d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:22.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.551+0000 7fef43fff700 1 --2- 172.21.15.184:0/1515706769 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fef440f51a0 0x7fef441054d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:22.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.551+0000 7fef43fff700 1 -- 172.21.15.184:0/1515706769 >> 172.21.15.184:0/1515706769 conn(0x7fef440f09a0 msgr2=0x7fef440f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:22.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.552+0000 7fef43fff700 1 -- 172.21.15.184:0/1515706769 shutdown_connections 2024-02-19T22:19:22.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:22.552+0000 7fef43fff700 1 -- 172.21.15.184:0/1515706769 wait complete. 2024-02-19T22:19:22.555 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:23.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:22 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:22.551143+0000 mon.smithi176 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.184:0/1515706769' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:24.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:22.401339+0000 mgr.smithi176.faoogw (mgr.14162) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:24.349 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:24.350 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:26.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:24.401638+0000 mgr.smithi176.faoogw (mgr.14162) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:26.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.179+0000 7f0811764700 1 -- 172.21.15.184:0/1292313556 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f080c0f51a0 msgr2=0x7f080c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:26.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.179+0000 7f0811764700 1 --2- 172.21.15.184:0/1292313556 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f080c0f51a0 0x7f080c0f55c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f07fc009fb0 tx=0x7f07fc009b30).stop 2024-02-19T22:19:26.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.179+0000 7f0811764700 1 -- 172.21.15.184:0/1292313556 shutdown_connections 2024-02-19T22:19:26.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.179+0000 7f0811764700 1 --2- 172.21.15.184:0/1292313556 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f080c0f51a0 0x7f080c0f55c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:26.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.179+0000 7f0811764700 1 -- 172.21.15.184:0/1292313556 >> 172.21.15.184:0/1292313556 conn(0x7f080c0f09a0 msgr2=0x7f080c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:26.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.179+0000 7f0811764700 1 -- 172.21.15.184:0/1292313556 shutdown_connections 2024-02-19T22:19:26.181 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.179+0000 7f0811764700 1 -- 172.21.15.184:0/1292313556 wait complete. 2024-02-19T22:19:26.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.179+0000 7f0811764700 1 Processor -- start 2024-02-19T22:19:26.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.180+0000 7f0811764700 1 -- start start 2024-02-19T22:19:26.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.180+0000 7f0811764700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f080c0f51a0 0x7f080c1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:26.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.180+0000 7f0811764700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c1059c0 con 0x7f080c0f51a0 2024-02-19T22:19:26.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.180+0000 7f080bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f080c0f51a0 0x7f080c1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:26.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.180+0000 7f080bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f080c0f51a0 0x7f080c1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:49402/0 (socket says 172.21.15.184:49402) 2024-02-19T22:19:26.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.180+0000 7f080bfff700 1 -- 172.21.15.184:0/649261394 learned_addr learned my addr 172.21.15.184:0/649261394 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:26.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.180+0000 7f080bfff700 1 -- 172.21.15.184:0/649261394 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07fc005040 con 0x7f080c0f51a0 2024-02-19T22:19:26.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.180+0000 7f080bfff700 1 --2- 172.21.15.184:0/649261394 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f080c0f51a0 0x7f080c1054d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f07fc0089f0 tx=0x7f07fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:26.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.181+0000 7f0808ff9700 1 -- 172.21.15.184:0/649261394 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07fc009320 con 0x7f080c0f51a0 2024-02-19T22:19:26.183 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.181+0000 7f0811764700 1 -- 172.21.15.184:0/649261394 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080c105bc0 con 0x7f080c0f51a0 2024-02-19T22:19:26.184 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.181+0000 7f0808ff9700 1 -- 172.21.15.184:0/649261394 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07fc013380 con 0x7f080c0f51a0 2024-02-19T22:19:26.184 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.181+0000 7f0808ff9700 1 -- 172.21.15.184:0/649261394 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07fc014e20 con 0x7f080c0f51a0 2024-02-19T22:19:26.184 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.181+0000 7f0811764700 1 -- 172.21.15.184:0/649261394 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080c106050 con 0x7f080c0f51a0 2024-02-19T22:19:26.185 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.181+0000 7f0808ff9700 1 -- 172.21.15.184:0/649261394 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07fc0095f0 con 0x7f080c0f51a0 2024-02-19T22:19:26.185 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.181+0000 7f0811764700 1 -- 172.21.15.184:0/649261394 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080c045ad0 con 0x7f080c0f51a0 2024-02-19T22:19:26.185 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.182+0000 7f0808ff9700 1 --2- 172.21.15.184:0/649261394 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f07f402ff10 0x7f07f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:26.185 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.182+0000 7f0808ff9700 1 -- 172.21.15.184:0/649261394 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07fc040f50 con 0x7f080c0f51a0 2024-02-19T22:19:26.185 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.182+0000 7f080b7fe700 1 --2- 172.21.15.184:0/649261394 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f07f402ff10 0x7f07f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:26.185 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.182+0000 7f080b7fe700 1 --2- 172.21.15.184:0/649261394 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f07f402ff10 0x7f07f40323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f08000096a0 tx=0x7f0800006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:26.188 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.186+0000 7f0808ff9700 1 -- 172.21.15.184:0/649261394 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07fc014420 con 0x7f080c0f51a0 2024-02-19T22:19:26.392 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.390+0000 7f0811764700 1 -- 172.21.15.184:0/649261394 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f080c106f00 con 0x7f080c0f51a0 2024-02-19T22:19:26.393 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.391+0000 7f0808ff9700 1 -- 172.21.15.184:0/649261394 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f07fc01d980 con 0x7f080c0f51a0 2024-02-19T22:19:26.393 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:26.393 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:26.394 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.392+0000 7f0811764700 1 -- 172.21.15.184:0/649261394 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f07f402ff10 msgr2=0x7f07f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:26.394 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.392+0000 7f0811764700 1 --2- 172.21.15.184:0/649261394 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f07f402ff10 0x7f07f40323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f08000096a0 tx=0x7f0800006b40).stop 2024-02-19T22:19:26.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.392+0000 7f0811764700 1 -- 172.21.15.184:0/649261394 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f080c0f51a0 msgr2=0x7f080c1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:26.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.392+0000 7f0811764700 1 --2- 172.21.15.184:0/649261394 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f080c0f51a0 0x7f080c1054d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f07fc0089f0 tx=0x7f07fc00b040).stop 2024-02-19T22:19:26.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.392+0000 7f0811764700 1 -- 172.21.15.184:0/649261394 shutdown_connections 2024-02-19T22:19:26.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.392+0000 7f0811764700 1 --2- 172.21.15.184:0/649261394 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f07f402ff10 0x7f07f40323d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:26.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.392+0000 7f0811764700 1 --2- 172.21.15.184:0/649261394 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f080c0f51a0 0x7f080c1054d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:26.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.392+0000 7f0811764700 1 -- 172.21.15.184:0/649261394 >> 172.21.15.184:0/649261394 conn(0x7f080c0f09a0 msgr2=0x7f080c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:26.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.392+0000 7f0811764700 1 -- 172.21.15.184:0/649261394 shutdown_connections 2024-02-19T22:19:26.395 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:26.392+0000 7f0811764700 1 -- 172.21.15.184:0/649261394 wait complete. 2024-02-19T22:19:26.395 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:27.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:26 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:26.392013+0000 mon.smithi176 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.184:0/649261394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:28.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:28.121 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:28.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:26.402013+0000 mgr.smithi176.faoogw (mgr.14162) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:30.041 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.039+0000 7f826bfff700 1 -- 172.21.15.184:0/869989777 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f826c060140 msgr2=0x7f826c0f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:30.041 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.039+0000 7f826bfff700 1 --2- 172.21.15.184:0/869989777 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f826c060140 0x7f826c0f8fc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f825c009fb0 tx=0x7f825c009b30).stop 2024-02-19T22:19:30.041 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.039+0000 7f826bfff700 1 -- 172.21.15.184:0/869989777 shutdown_connections 2024-02-19T22:19:30.041 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.039+0000 7f826bfff700 1 --2- 172.21.15.184:0/869989777 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f826c060140 0x7f826c0f8fc0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:30.041 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.039+0000 7f826bfff700 1 -- 172.21.15.184:0/869989777 >> 172.21.15.184:0/869989777 conn(0x7f826c0f09a0 msgr2=0x7f826c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:30.041 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.039+0000 7f826bfff700 1 -- 172.21.15.184:0/869989777 shutdown_connections 2024-02-19T22:19:30.042 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.039+0000 7f826bfff700 1 -- 172.21.15.184:0/869989777 wait complete. 2024-02-19T22:19:30.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.040+0000 7f826bfff700 1 Processor -- start 2024-02-19T22:19:30.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.040+0000 7f826bfff700 1 -- start start 2024-02-19T22:19:30.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.040+0000 7f826bfff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f826c060140 0x7f826c106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:30.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.040+0000 7f826bfff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c106af0 con 0x7f826c060140 2024-02-19T22:19:30.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.040+0000 7f826affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f826c060140 0x7f826c106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:30.043 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.040+0000 7f826affd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f826c060140 0x7f826c106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:37124/0 (socket says 172.21.15.184:37124) 2024-02-19T22:19:30.044 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.040+0000 7f826affd700 1 -- 172.21.15.184:0/2577875636 learned_addr learned my addr 172.21.15.184:0/2577875636 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:30.044 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.040+0000 7f826affd700 1 -- 172.21.15.184:0/2577875636 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c005040 con 0x7f826c060140 2024-02-19T22:19:30.044 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.041+0000 7f826affd700 1 --2- 172.21.15.184:0/2577875636 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f826c060140 0x7f826c106600 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f825c005010 tx=0x7f825c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:30.044 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.041+0000 7f82537fe700 1 -- 172.21.15.184:0/2577875636 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f825c009320 con 0x7f826c060140 2024-02-19T22:19:30.044 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.041+0000 7f826bfff700 1 -- 172.21.15.184:0/2577875636 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c106cf0 con 0x7f826c060140 2024-02-19T22:19:30.044 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.041+0000 7f826bfff700 1 -- 172.21.15.184:0/2577875636 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c107180 con 0x7f826c060140 2024-02-19T22:19:30.044 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.041+0000 7f82537fe700 1 -- 172.21.15.184:0/2577875636 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f825c013380 con 0x7f826c060140 2024-02-19T22:19:30.044 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.041+0000 7f82537fe700 1 -- 172.21.15.184:0/2577875636 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f825c014df0 con 0x7f826c060140 2024-02-19T22:19:30.045 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.042+0000 7f82537fe700 1 -- 172.21.15.184:0/2577875636 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f825c0263a0 con 0x7f826c060140 2024-02-19T22:19:30.045 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.042+0000 7f826bfff700 1 -- 172.21.15.184:0/2577875636 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826c045ad0 con 0x7f826c060140 2024-02-19T22:19:30.045 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.042+0000 7f82537fe700 1 --2- 172.21.15.184:0/2577875636 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f825402ff10 0x7f82540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:30.045 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.042+0000 7f82537fe700 1 -- 172.21.15.184:0/2577875636 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f825c041180 con 0x7f826c060140 2024-02-19T22:19:30.045 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.042+0000 7f826a7fc700 1 --2- 172.21.15.184:0/2577875636 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f825402ff10 0x7f82540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:30.045 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.042+0000 7f826a7fc700 1 --2- 172.21.15.184:0/2577875636 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f825402ff10 0x7f82540323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f82600096a0 tx=0x7f8260006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:30.048 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.047+0000 7f82537fe700 1 -- 172.21.15.184:0/2577875636 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f825c012330 con 0x7f826c060140 2024-02-19T22:19:30.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:29 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:28.402317+0000 mgr.smithi176.faoogw (mgr.14162) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:30.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.251+0000 7f826bfff700 1 -- 172.21.15.184:0/2577875636 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f826c108030 con 0x7f826c060140 2024-02-19T22:19:30.254 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.252+0000 7f82537fe700 1 -- 172.21.15.184:0/2577875636 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f825c01ce90 con 0x7f826c060140 2024-02-19T22:19:30.255 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:30.255 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:30.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.254+0000 7f826bfff700 1 -- 172.21.15.184:0/2577875636 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f825402ff10 msgr2=0x7f82540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:30.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.254+0000 7f826bfff700 1 --2- 172.21.15.184:0/2577875636 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f825402ff10 0x7f82540323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f82600096a0 tx=0x7f8260006b40).stop 2024-02-19T22:19:30.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.254+0000 7f826bfff700 1 -- 172.21.15.184:0/2577875636 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f826c060140 msgr2=0x7f826c106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:30.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.254+0000 7f826bfff700 1 --2- 172.21.15.184:0/2577875636 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f826c060140 0x7f826c106600 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f825c005010 tx=0x7f825c00b040).stop 2024-02-19T22:19:30.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.254+0000 7f826bfff700 1 -- 172.21.15.184:0/2577875636 shutdown_connections 2024-02-19T22:19:30.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.254+0000 7f826bfff700 1 --2- 172.21.15.184:0/2577875636 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f825402ff10 0x7f82540323d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:30.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.254+0000 7f826bfff700 1 --2- 172.21.15.184:0/2577875636 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f826c060140 0x7f826c106600 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:30.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.254+0000 7f826bfff700 1 -- 172.21.15.184:0/2577875636 >> 172.21.15.184:0/2577875636 conn(0x7f826c0f09a0 msgr2=0x7f826c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:30.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.254+0000 7f826bfff700 1 -- 172.21.15.184:0/2577875636 shutdown_connections 2024-02-19T22:19:30.257 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:30.254+0000 7f826bfff700 1 -- 172.21.15.184:0/2577875636 wait complete. 2024-02-19T22:19:30.257 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:31.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:30 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:30.253613+0000 mon.smithi176 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.184:0/2577875636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:32.062 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:32.063 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:32.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:30.402725+0000 mgr.smithi176.faoogw (mgr.14162) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:33.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.852+0000 7f61f3f73700 1 -- 172.21.15.184:0/302865895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61ec0f5410 msgr2=0x7f61ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:33.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.852+0000 7f61f3f73700 1 --2- 172.21.15.184:0/302865895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61ec0f5410 0x7f61ec0f5830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f61e8009fb0 tx=0x7f61e8009b30).stop 2024-02-19T22:19:33.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.852+0000 7f61f3f73700 1 -- 172.21.15.184:0/302865895 shutdown_connections 2024-02-19T22:19:33.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.852+0000 7f61f3f73700 1 --2- 172.21.15.184:0/302865895 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61ec0f5410 0x7f61ec0f5830 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:33.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.852+0000 7f61f3f73700 1 -- 172.21.15.184:0/302865895 >> 172.21.15.184:0/302865895 conn(0x7f61ec0f0b70 msgr2=0x7f61ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:33.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.852+0000 7f61f3f73700 1 -- 172.21.15.184:0/302865895 shutdown_connections 2024-02-19T22:19:33.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.852+0000 7f61f3f73700 1 -- 172.21.15.184:0/302865895 wait complete. 2024-02-19T22:19:33.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.852+0000 7f61f3f73700 1 Processor -- start 2024-02-19T22:19:33.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.853+0000 7f61f3f73700 1 -- start start 2024-02-19T22:19:33.889 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.853+0000 7f61f3f73700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61ec0f5410 0x7f61ec1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:33.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.853+0000 7f61f3f73700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec106cc0 con 0x7f61ec0f5410 2024-02-19T22:19:33.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.853+0000 7f61f1d0f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61ec0f5410 0x7f61ec1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:33.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.853+0000 7f61f1d0f700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61ec0f5410 0x7f61ec1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:37132/0 (socket says 172.21.15.184:37132) 2024-02-19T22:19:33.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.853+0000 7f61f1d0f700 1 -- 172.21.15.184:0/897125868 learned_addr learned my addr 172.21.15.184:0/897125868 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:33.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.853+0000 7f61f1d0f700 1 -- 172.21.15.184:0/897125868 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e8005040 con 0x7f61ec0f5410 2024-02-19T22:19:33.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.853+0000 7f61f1d0f700 1 --2- 172.21.15.184:0/897125868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61ec0f5410 0x7f61ec1067d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f61e8005010 tx=0x7f61e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:33.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.853+0000 7f61e27fc700 1 -- 172.21.15.184:0/897125868 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61e8009320 con 0x7f61ec0f5410 2024-02-19T22:19:33.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.854+0000 7f61e27fc700 1 -- 172.21.15.184:0/897125868 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f61e8013380 con 0x7f61ec0f5410 2024-02-19T22:19:33.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.854+0000 7f61f3f73700 1 -- 172.21.15.184:0/897125868 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ec106ec0 con 0x7f61ec0f5410 2024-02-19T22:19:33.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.854+0000 7f61e27fc700 1 -- 172.21.15.184:0/897125868 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61e8014e50 con 0x7f61ec0f5410 2024-02-19T22:19:33.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.854+0000 7f61f3f73700 1 -- 172.21.15.184:0/897125868 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ec107350 con 0x7f61ec0f5410 2024-02-19T22:19:33.890 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.854+0000 7f61e27fc700 1 -- 172.21.15.184:0/897125868 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61e8014420 con 0x7f61ec0f5410 2024-02-19T22:19:33.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.854+0000 7f61f3f73700 1 -- 172.21.15.184:0/897125868 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ec045ad0 con 0x7f61ec0f5410 2024-02-19T22:19:33.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.854+0000 7f61e27fc700 1 --2- 172.21.15.184:0/897125868 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f61d802ff10 0x7f61d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:33.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.855+0000 7f61e27fc700 1 -- 172.21.15.184:0/897125868 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61e8040e30 con 0x7f61ec0f5410 2024-02-19T22:19:33.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.855+0000 7f61f150e700 1 --2- 172.21.15.184:0/897125868 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f61d802ff10 0x7f61d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:33.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.855+0000 7f61f150e700 1 --2- 172.21.15.184:0/897125868 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f61d802ff10 0x7f61d80323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f61dc0096a0 tx=0x7f61dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:33.891 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:33.859+0000 7f61e27fc700 1 -- 172.21.15.184:0/897125868 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61e80095f0 con 0x7f61ec0f5410 2024-02-19T22:19:34.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:34.063+0000 7f61f3f73700 1 -- 172.21.15.184:0/897125868 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61ec108200 con 0x7f61ec0f5410 2024-02-19T22:19:34.066 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:34.064+0000 7f61e27fc700 1 -- 172.21.15.184:0/897125868 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f61e801db70 con 0x7f61ec0f5410 2024-02-19T22:19:34.066 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:34.066 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:34.068 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:34.065+0000 7f61f3f73700 1 -- 172.21.15.184:0/897125868 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f61d802ff10 msgr2=0x7f61d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:34.068 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:34.065+0000 7f61f3f73700 1 --2- 172.21.15.184:0/897125868 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f61d802ff10 0x7f61d80323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f61dc0096a0 tx=0x7f61dc006b40).stop 2024-02-19T22:19:34.068 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:34.065+0000 7f61f3f73700 1 -- 172.21.15.184:0/897125868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61ec0f5410 msgr2=0x7f61ec1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:34.068 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:34.065+0000 7f61f3f73700 1 --2- 172.21.15.184:0/897125868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61ec0f5410 0x7f61ec1067d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f61e8005010 tx=0x7f61e800b040).stop 2024-02-19T22:19:34.068 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:34.065+0000 7f61f3f73700 1 -- 172.21.15.184:0/897125868 shutdown_connections 2024-02-19T22:19:34.068 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:34.065+0000 7f61f3f73700 1 --2- 172.21.15.184:0/897125868 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f61d802ff10 0x7f61d80323d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:34.068 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:34.065+0000 7f61f3f73700 1 --2- 172.21.15.184:0/897125868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61ec0f5410 0x7f61ec1067d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:34.068 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:34.065+0000 7f61f3f73700 1 -- 172.21.15.184:0/897125868 >> 172.21.15.184:0/897125868 conn(0x7f61ec0f0b70 msgr2=0x7f61ec05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:34.068 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:34.065+0000 7f61f3f73700 1 -- 172.21.15.184:0/897125868 shutdown_connections 2024-02-19T22:19:34.068 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:34.065+0000 7f61f3f73700 1 -- 172.21.15.184:0/897125868 wait complete. 2024-02-19T22:19:34.069 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:34.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:32.403043+0000 mgr.smithi176.faoogw (mgr.14162) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:35.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:34 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:34.065087+0000 mon.smithi176 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.184:0/897125868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:35.768 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:35.768 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:36.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:34.403488+0000 mgr.smithi176.faoogw (mgr.14162) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:37.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:35.976849+0000 mon.smithi176 (mon.0) 320 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:19:37.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:35.978111+0000 mon.smithi176 (mon.0) 321 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:19:37.848 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.845+0000 7f815bc94700 1 -- 172.21.15.184:0/4177631089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81540f5410 msgr2=0x7f81540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:37.848 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.845+0000 7f815bc94700 1 --2- 172.21.15.184:0/4177631089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81540f5410 0x7f81540f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f8144009fb0 tx=0x7f8144009b30).stop 2024-02-19T22:19:37.848 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.845+0000 7f815bc94700 1 -- 172.21.15.184:0/4177631089 shutdown_connections 2024-02-19T22:19:37.848 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.845+0000 7f815bc94700 1 --2- 172.21.15.184:0/4177631089 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81540f5410 0x7f81540f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:37.848 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.845+0000 7f815bc94700 1 -- 172.21.15.184:0/4177631089 >> 172.21.15.184:0/4177631089 conn(0x7f81540f0b70 msgr2=0x7f81540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:37.848 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.845+0000 7f815bc94700 1 -- 172.21.15.184:0/4177631089 shutdown_connections 2024-02-19T22:19:37.848 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.845+0000 7f815bc94700 1 -- 172.21.15.184:0/4177631089 wait complete. 2024-02-19T22:19:37.848 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.846+0000 7f815bc94700 1 Processor -- start 2024-02-19T22:19:37.848 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.846+0000 7f815bc94700 1 -- start start 2024-02-19T22:19:37.849 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.846+0000 7f815bc94700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81540f5410 0x7f81541056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:37.849 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.846+0000 7f815bc94700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8154105bd0 con 0x7f81540f5410 2024-02-19T22:19:37.849 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.846+0000 7f8159a30700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81540f5410 0x7f81541056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:37.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.846+0000 7f8159a30700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81540f5410 0x7f81541056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:34606/0 (socket says 172.21.15.184:34606) 2024-02-19T22:19:37.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.846+0000 7f8159a30700 1 -- 172.21.15.184:0/3187620328 learned_addr learned my addr 172.21.15.184:0/3187620328 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:37.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.847+0000 7f8159a30700 1 -- 172.21.15.184:0/3187620328 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8144005040 con 0x7f81540f5410 2024-02-19T22:19:37.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.847+0000 7f8159a30700 1 --2- 172.21.15.184:0/3187620328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81540f5410 0x7f81541056e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f81440089f0 tx=0x7f814400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:37.850 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.848+0000 7f814a7fc700 1 -- 172.21.15.184:0/3187620328 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8144009320 con 0x7f81540f5410 2024-02-19T22:19:37.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.848+0000 7f814a7fc700 1 -- 172.21.15.184:0/3187620328 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8144013380 con 0x7f81540f5410 2024-02-19T22:19:37.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.848+0000 7f815bc94700 1 -- 172.21.15.184:0/3187620328 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8154105dd0 con 0x7f81540f5410 2024-02-19T22:19:37.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.848+0000 7f814a7fc700 1 -- 172.21.15.184:0/3187620328 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f814401ed80 con 0x7f81540f5410 2024-02-19T22:19:37.851 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.848+0000 7f815bc94700 1 -- 172.21.15.184:0/3187620328 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8154106260 con 0x7f81540f5410 2024-02-19T22:19:37.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.849+0000 7f814a7fc700 1 -- 172.21.15.184:0/3187620328 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f814401e420 con 0x7f81540f5410 2024-02-19T22:19:37.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.849+0000 7f815bc94700 1 -- 172.21.15.184:0/3187620328 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8154045ad0 con 0x7f81540f5410 2024-02-19T22:19:37.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.849+0000 7f814a7fc700 1 --2- 172.21.15.184:0/3187620328 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f814002ff60 0x7f8140032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:37.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.849+0000 7f814a7fc700 1 -- 172.21.15.184:0/3187620328 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8144041150 con 0x7f81540f5410 2024-02-19T22:19:37.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.849+0000 7f815922f700 1 --2- 172.21.15.184:0/3187620328 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f814002ff60 0x7f8140032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:37.852 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.850+0000 7f815922f700 1 --2- 172.21.15.184:0/3187620328 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f814002ff60 0x7f8140032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f81500096a0 tx=0x7f8150006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:37.856 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:37.854+0000 7f814a7fc700 1 -- 172.21.15.184:0/3187620328 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81440095f0 con 0x7f81540f5410 2024-02-19T22:19:38.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:38.060+0000 7f815bc94700 1 -- 172.21.15.184:0/3187620328 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8154107110 con 0x7f81540f5410 2024-02-19T22:19:38.063 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:38.060+0000 7f814a7fc700 1 -- 172.21.15.184:0/3187620328 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8144004490 con 0x7f81540f5410 2024-02-19T22:19:38.063 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:38.063 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:38.064 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:38.062+0000 7f815bc94700 1 -- 172.21.15.184:0/3187620328 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f814002ff60 msgr2=0x7f8140032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:38.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:38.062+0000 7f815bc94700 1 --2- 172.21.15.184:0/3187620328 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f814002ff60 0x7f8140032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f81500096a0 tx=0x7f8150006b40).stop 2024-02-19T22:19:38.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:38.062+0000 7f815bc94700 1 -- 172.21.15.184:0/3187620328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81540f5410 msgr2=0x7f81541056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:38.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:38.062+0000 7f815bc94700 1 --2- 172.21.15.184:0/3187620328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81540f5410 0x7f81541056e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f81440089f0 tx=0x7f814400b040).stop 2024-02-19T22:19:38.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:38.062+0000 7f815bc94700 1 -- 172.21.15.184:0/3187620328 shutdown_connections 2024-02-19T22:19:38.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:38.062+0000 7f815bc94700 1 --2- 172.21.15.184:0/3187620328 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f814002ff60 0x7f8140032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:38.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:38.062+0000 7f815bc94700 1 --2- 172.21.15.184:0/3187620328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f81540f5410 0x7f81541056e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:38.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:38.062+0000 7f815bc94700 1 -- 172.21.15.184:0/3187620328 >> 172.21.15.184:0/3187620328 conn(0x7f81540f0b70 msgr2=0x7f81540f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:38.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:38.062+0000 7f815bc94700 1 -- 172.21.15.184:0/3187620328 shutdown_connections 2024-02-19T22:19:38.065 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:38.062+0000 7f815bc94700 1 -- 172.21.15.184:0/3187620328 wait complete. 2024-02-19T22:19:38.066 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:38.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:36.403866+0000 mgr.smithi176.faoogw (mgr.14162) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:39.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:38 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:38.061941+0000 mon.smithi176 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.184:0/3187620328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:39.685 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:39.686 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:40.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:38.404175+0000 mgr.smithi176.faoogw (mgr.14162) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:41.553 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.551+0000 7f64e7aff700 1 -- 172.21.15.184:0/3139519761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e00f5410 msgr2=0x7f64e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:41.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.551+0000 7f64e7aff700 1 --2- 172.21.15.184:0/3139519761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e00f5410 0x7f64e00f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f64d0009fb0 tx=0x7f64d0009b30).stop 2024-02-19T22:19:41.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.551+0000 7f64e7aff700 1 -- 172.21.15.184:0/3139519761 shutdown_connections 2024-02-19T22:19:41.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.551+0000 7f64e7aff700 1 --2- 172.21.15.184:0/3139519761 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e00f5410 0x7f64e00f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:41.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.551+0000 7f64e7aff700 1 -- 172.21.15.184:0/3139519761 >> 172.21.15.184:0/3139519761 conn(0x7f64e00f0b70 msgr2=0x7f64e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:41.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.551+0000 7f64e7aff700 1 -- 172.21.15.184:0/3139519761 shutdown_connections 2024-02-19T22:19:41.554 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.552+0000 7f64e7aff700 1 -- 172.21.15.184:0/3139519761 wait complete. 2024-02-19T22:19:41.555 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.552+0000 7f64e7aff700 1 Processor -- start 2024-02-19T22:19:41.555 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.552+0000 7f64e7aff700 1 -- start start 2024-02-19T22:19:41.555 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.552+0000 7f64e7aff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e00f5410 0x7f64e00fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:41.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.552+0000 7f64e7aff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e00fb960 con 0x7f64e00f5410 2024-02-19T22:19:41.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.552+0000 7f64e589b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e00f5410 0x7f64e00fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:41.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.552+0000 7f64e589b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e00f5410 0x7f64e00fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:34616/0 (socket says 172.21.15.184:34616) 2024-02-19T22:19:41.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.552+0000 7f64e589b700 1 -- 172.21.15.184:0/2655192095 learned_addr learned my addr 172.21.15.184:0/2655192095 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:41.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.553+0000 7f64e589b700 1 -- 172.21.15.184:0/2655192095 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d0005040 con 0x7f64e00f5410 2024-02-19T22:19:41.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.553+0000 7f64e589b700 1 --2- 172.21.15.184:0/2655192095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e00f5410 0x7f64e00fd4e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f64d0005010 tx=0x7f64d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:41.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.553+0000 7f64d67fc700 1 -- 172.21.15.184:0/2655192095 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64d0009320 con 0x7f64e00f5410 2024-02-19T22:19:41.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.553+0000 7f64e7aff700 1 -- 172.21.15.184:0/2655192095 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e00fbb00 con 0x7f64e00f5410 2024-02-19T22:19:41.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.553+0000 7f64d67fc700 1 -- 172.21.15.184:0/2655192095 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f64d0013380 con 0x7f64e00f5410 2024-02-19T22:19:41.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.553+0000 7f64d67fc700 1 -- 172.21.15.184:0/2655192095 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64d0014e50 con 0x7f64e00f5410 2024-02-19T22:19:41.556 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.553+0000 7f64e7aff700 1 -- 172.21.15.184:0/2655192095 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e00fbf90 con 0x7f64e00f5410 2024-02-19T22:19:41.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.554+0000 7f64d67fc700 1 -- 172.21.15.184:0/2655192095 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64d00095f0 con 0x7f64e00f5410 2024-02-19T22:19:41.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.554+0000 7f64e7aff700 1 -- 172.21.15.184:0/2655192095 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e0059100 con 0x7f64e00f5410 2024-02-19T22:19:41.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.554+0000 7f64d67fc700 1 --2- 172.21.15.184:0/2655192095 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f64cc02ff10 0x7f64cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:41.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.554+0000 7f64d67fc700 1 -- 172.21.15.184:0/2655192095 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64d0040e90 con 0x7f64e00f5410 2024-02-19T22:19:41.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.554+0000 7f64e509a700 1 --2- 172.21.15.184:0/2655192095 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f64cc02ff10 0x7f64cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:41.557 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.555+0000 7f64e509a700 1 --2- 172.21.15.184:0/2655192095 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f64cc02ff10 0x7f64cc0323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f64dc0096a0 tx=0x7f64dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:41.561 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.559+0000 7f64d67fc700 1 -- 172.21.15.184:0/2655192095 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64d0014420 con 0x7f64e00f5410 2024-02-19T22:19:41.766 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.764+0000 7f64e7aff700 1 -- 172.21.15.184:0/2655192095 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64e00fafa0 con 0x7f64e00f5410 2024-02-19T22:19:41.766 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.764+0000 7f64d67fc700 1 -- 172.21.15.184:0/2655192095 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f64d0010230 con 0x7f64e00f5410 2024-02-19T22:19:41.766 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:41.767 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:41.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.766+0000 7f64e7aff700 1 -- 172.21.15.184:0/2655192095 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f64cc02ff10 msgr2=0x7f64cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:41.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.766+0000 7f64e7aff700 1 --2- 172.21.15.184:0/2655192095 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f64cc02ff10 0x7f64cc0323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f64dc0096a0 tx=0x7f64dc006b40).stop 2024-02-19T22:19:41.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.766+0000 7f64e7aff700 1 -- 172.21.15.184:0/2655192095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e00f5410 msgr2=0x7f64e00fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:41.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.766+0000 7f64e7aff700 1 --2- 172.21.15.184:0/2655192095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e00f5410 0x7f64e00fd4e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f64d0005010 tx=0x7f64d000b040).stop 2024-02-19T22:19:41.768 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.766+0000 7f64e7aff700 1 -- 172.21.15.184:0/2655192095 shutdown_connections 2024-02-19T22:19:41.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.766+0000 7f64e7aff700 1 --2- 172.21.15.184:0/2655192095 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f64cc02ff10 0x7f64cc0323d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:41.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.766+0000 7f64e7aff700 1 --2- 172.21.15.184:0/2655192095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64e00f5410 0x7f64e00fd4e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:41.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.766+0000 7f64e7aff700 1 -- 172.21.15.184:0/2655192095 >> 172.21.15.184:0/2655192095 conn(0x7f64e00f0b70 msgr2=0x7f64e005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:41.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.766+0000 7f64e7aff700 1 -- 172.21.15.184:0/2655192095 shutdown_connections 2024-02-19T22:19:41.769 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:41.766+0000 7f64e7aff700 1 -- 172.21.15.184:0/2655192095 wait complete. 2024-02-19T22:19:41.769 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:42.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:40.404581+0000 mgr.smithi176.faoogw (mgr.14162) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:42.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:41.765731+0000 mon.smithi176 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.184:0/2655192095' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:43.411 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:43.411 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:44.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:42.404973+0000 mgr.smithi176.faoogw (mgr.14162) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:45.380 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.378+0000 7f5879627700 1 -- 172.21.15.184:0/2942261905 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58740f5410 msgr2=0x7f58740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:45.380 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.378+0000 7f5879627700 1 --2- 172.21.15.184:0/2942261905 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58740f5410 0x7f58740f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f585c009fb0 tx=0x7f585c009b30).stop 2024-02-19T22:19:45.380 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.378+0000 7f5879627700 1 -- 172.21.15.184:0/2942261905 shutdown_connections 2024-02-19T22:19:45.380 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.378+0000 7f5879627700 1 --2- 172.21.15.184:0/2942261905 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58740f5410 0x7f58740f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:45.380 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.378+0000 7f5879627700 1 -- 172.21.15.184:0/2942261905 >> 172.21.15.184:0/2942261905 conn(0x7f58740f0b70 msgr2=0x7f58740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:45.381 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.378+0000 7f5879627700 1 -- 172.21.15.184:0/2942261905 shutdown_connections 2024-02-19T22:19:45.381 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.378+0000 7f5879627700 1 -- 172.21.15.184:0/2942261905 wait complete. 2024-02-19T22:19:45.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.378+0000 7f5879627700 1 Processor -- start 2024-02-19T22:19:45.382 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.378+0000 7f5879627700 1 -- start start 2024-02-19T22:19:45.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.379+0000 7f5879627700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58740f5410 0x7f5874106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:45.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.379+0000 7f5879627700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5874106d00 con 0x7f58740f5410 2024-02-19T22:19:45.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.379+0000 7f5872ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58740f5410 0x7f5874106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:45.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.379+0000 7f5872ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58740f5410 0x7f5874106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:34642/0 (socket says 172.21.15.184:34642) 2024-02-19T22:19:45.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.379+0000 7f5872ffd700 1 -- 172.21.15.184:0/806632260 learned_addr learned my addr 172.21.15.184:0/806632260 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:45.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.379+0000 7f5872ffd700 1 -- 172.21.15.184:0/806632260 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585c005040 con 0x7f58740f5410 2024-02-19T22:19:45.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.379+0000 7f5872ffd700 1 --2- 172.21.15.184:0/806632260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58740f5410 0x7f5874106810 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f585c005010 tx=0x7f585c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:45.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.379+0000 7f586b7fe700 1 -- 172.21.15.184:0/806632260 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f585c009320 con 0x7f58740f5410 2024-02-19T22:19:45.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.379+0000 7f586b7fe700 1 -- 172.21.15.184:0/806632260 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f585c013380 con 0x7f58740f5410 2024-02-19T22:19:45.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.379+0000 7f5879627700 1 -- 172.21.15.184:0/806632260 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5874106f00 con 0x7f58740f5410 2024-02-19T22:19:45.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.379+0000 7f586b7fe700 1 -- 172.21.15.184:0/806632260 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f585c014e50 con 0x7f58740f5410 2024-02-19T22:19:45.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.380+0000 7f5879627700 1 -- 172.21.15.184:0/806632260 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5874107390 con 0x7f58740f5410 2024-02-19T22:19:45.383 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.380+0000 7f586b7fe700 1 -- 172.21.15.184:0/806632260 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f585c014420 con 0x7f58740f5410 2024-02-19T22:19:45.384 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.381+0000 7f586b7fe700 1 --2- 172.21.15.184:0/806632260 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f586002ff10 0x7f58600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:45.384 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.381+0000 7f5879627700 1 -- 172.21.15.184:0/806632260 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5874045ad0 con 0x7f58740f5410 2024-02-19T22:19:45.384 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.381+0000 7f586b7fe700 1 -- 172.21.15.184:0/806632260 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f585c040f70 con 0x7f58740f5410 2024-02-19T22:19:45.384 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.381+0000 7f58727fc700 1 --2- 172.21.15.184:0/806632260 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f586002ff10 0x7f58600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:45.384 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.381+0000 7f58727fc700 1 --2- 172.21.15.184:0/806632260 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f586002ff10 0x7f58600323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f58640096a0 tx=0x7f5864006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:45.387 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.386+0000 7f586b7fe700 1 -- 172.21.15.184:0/806632260 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f585c0095f0 con 0x7f58740f5410 2024-02-19T22:19:45.591 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.589+0000 7f5879627700 1 -- 172.21.15.184:0/806632260 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58741081e0 con 0x7f58740f5410 2024-02-19T22:19:45.591 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.590+0000 7f586b7fe700 1 -- 172.21.15.184:0/806632260 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f585c02d090 con 0x7f58740f5410 2024-02-19T22:19:45.592 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:45.592 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:45.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.591+0000 7f5879627700 1 -- 172.21.15.184:0/806632260 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f586002ff10 msgr2=0x7f58600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:45.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.591+0000 7f5879627700 1 --2- 172.21.15.184:0/806632260 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f586002ff10 0x7f58600323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f58640096a0 tx=0x7f5864006b40).stop 2024-02-19T22:19:45.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.591+0000 7f5879627700 1 -- 172.21.15.184:0/806632260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58740f5410 msgr2=0x7f5874106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:45.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.591+0000 7f5879627700 1 --2- 172.21.15.184:0/806632260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58740f5410 0x7f5874106810 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f585c005010 tx=0x7f585c00b040).stop 2024-02-19T22:19:45.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.591+0000 7f5879627700 1 -- 172.21.15.184:0/806632260 shutdown_connections 2024-02-19T22:19:45.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.591+0000 7f5879627700 1 --2- 172.21.15.184:0/806632260 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f586002ff10 0x7f58600323d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:45.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.591+0000 7f5879627700 1 --2- 172.21.15.184:0/806632260 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f58740f5410 0x7f5874106810 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:45.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.591+0000 7f5879627700 1 -- 172.21.15.184:0/806632260 >> 172.21.15.184:0/806632260 conn(0x7f58740f0b70 msgr2=0x7f587405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:45.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.591+0000 7f5879627700 1 -- 172.21.15.184:0/806632260 shutdown_connections 2024-02-19T22:19:45.594 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:45.591+0000 7f5879627700 1 -- 172.21.15.184:0/806632260 wait complete. 2024-02-19T22:19:45.595 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:46.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:44.405319+0000 mgr.smithi176.faoogw (mgr.14162) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:46.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:45 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:45.591059+0000 mon.smithi176 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.184:0/806632260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:47.307 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:47.307 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:48.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:47 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:46.405688+0000 mgr.smithi176.faoogw (mgr.14162) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:49.207 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.202+0000 7fb2301c6700 1 -- 172.21.15.184:0/2453075133 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2280f6a80 msgr2=0x7fb2280f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:49.207 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.202+0000 7fb2301c6700 1 --2- 172.21.15.184:0/2453075133 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2280f6a80 0x7fb2280f6ea0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fb224009fb0 tx=0x7fb224009b30).stop 2024-02-19T22:19:49.207 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.203+0000 7fb2301c6700 1 -- 172.21.15.184:0/2453075133 shutdown_connections 2024-02-19T22:19:49.207 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.203+0000 7fb2301c6700 1 --2- 172.21.15.184:0/2453075133 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2280f6a80 0x7fb2280f6ea0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:49.207 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.203+0000 7fb2301c6700 1 -- 172.21.15.184:0/2453075133 >> 172.21.15.184:0/2453075133 conn(0x7fb2280f02e0 msgr2=0x7fb2280f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:49.207 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.203+0000 7fb2301c6700 1 -- 172.21.15.184:0/2453075133 shutdown_connections 2024-02-19T22:19:49.207 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.203+0000 7fb2301c6700 1 -- 172.21.15.184:0/2453075133 wait complete. 2024-02-19T22:19:49.207 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.203+0000 7fb2301c6700 1 Processor -- start 2024-02-19T22:19:49.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.203+0000 7fb2301c6700 1 -- start start 2024-02-19T22:19:49.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.204+0000 7fb2301c6700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2280f6a80 0x7fb2281094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:49.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.204+0000 7fb2301c6700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2280f5330 con 0x7fb2280f6a80 2024-02-19T22:19:49.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.204+0000 7fb22df62700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2280f6a80 0x7fb2281094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:49.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.204+0000 7fb22df62700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2280f6a80 0x7fb2281094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:48456/0 (socket says 172.21.15.184:48456) 2024-02-19T22:19:49.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.204+0000 7fb22df62700 1 -- 172.21.15.184:0/1991641713 learned_addr learned my addr 172.21.15.184:0/1991641713 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:49.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.204+0000 7fb22df62700 1 -- 172.21.15.184:0/1991641713 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb224005040 con 0x7fb2280f6a80 2024-02-19T22:19:49.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.204+0000 7fb22df62700 1 --2- 172.21.15.184:0/1991641713 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2280f6a80 0x7fb2281094a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fb2240089f0 tx=0x7fb22400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:49.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.204+0000 7fb21a7fc700 1 -- 172.21.15.184:0/1991641713 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb224009320 con 0x7fb2280f6a80 2024-02-19T22:19:49.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.204+0000 7fb2301c6700 1 -- 172.21.15.184:0/1991641713 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2280f5530 con 0x7fb2280f6a80 2024-02-19T22:19:49.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.204+0000 7fb21a7fc700 1 -- 172.21.15.184:0/1991641713 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb224013380 con 0x7fb2280f6a80 2024-02-19T22:19:49.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.204+0000 7fb21a7fc700 1 -- 172.21.15.184:0/1991641713 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb224014e20 con 0x7fb2280f6a80 2024-02-19T22:19:49.208 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.204+0000 7fb2301c6700 1 -- 172.21.15.184:0/1991641713 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb228109b50 con 0x7fb2280f6a80 2024-02-19T22:19:49.209 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.205+0000 7fb21a7fc700 1 -- 172.21.15.184:0/1991641713 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2240095f0 con 0x7fb2280f6a80 2024-02-19T22:19:49.209 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.205+0000 7fb2301c6700 1 -- 172.21.15.184:0/1991641713 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb228045ad0 con 0x7fb2280f6a80 2024-02-19T22:19:49.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.205+0000 7fb21a7fc700 1 --2- 172.21.15.184:0/1991641713 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb21402ff10 0x7fb2140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:49.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.206+0000 7fb21a7fc700 1 -- 172.21.15.184:0/1991641713 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb224040e90 con 0x7fb2280f6a80 2024-02-19T22:19:49.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.206+0000 7fb22d761700 1 --2- 172.21.15.184:0/1991641713 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb21402ff10 0x7fb2140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:49.210 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.206+0000 7fb22d761700 1 --2- 172.21.15.184:0/1991641713 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb21402ff10 0x7fb2140323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fb21c0096a0 tx=0x7fb21c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:49.212 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.210+0000 7fb21a7fc700 1 -- 172.21.15.184:0/1991641713 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb224014420 con 0x7fb2280f6a80 2024-02-19T22:19:49.417 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.415+0000 7fb2301c6700 1 -- 172.21.15.184:0/1991641713 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb22810a370 con 0x7fb2280f6a80 2024-02-19T22:19:49.418 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.416+0000 7fb21a7fc700 1 -- 172.21.15.184:0/1991641713 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb22401d980 con 0x7fb2280f6a80 2024-02-19T22:19:49.418 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:49.418 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:49.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.417+0000 7fb2301c6700 1 -- 172.21.15.184:0/1991641713 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb21402ff10 msgr2=0x7fb2140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:49.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.417+0000 7fb2301c6700 1 --2- 172.21.15.184:0/1991641713 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb21402ff10 0x7fb2140323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fb21c0096a0 tx=0x7fb21c006b40).stop 2024-02-19T22:19:49.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.417+0000 7fb2301c6700 1 -- 172.21.15.184:0/1991641713 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2280f6a80 msgr2=0x7fb2281094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:49.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.417+0000 7fb2301c6700 1 --2- 172.21.15.184:0/1991641713 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2280f6a80 0x7fb2281094a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fb2240089f0 tx=0x7fb22400b040).stop 2024-02-19T22:19:49.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.417+0000 7fb2301c6700 1 -- 172.21.15.184:0/1991641713 shutdown_connections 2024-02-19T22:19:49.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.417+0000 7fb2301c6700 1 --2- 172.21.15.184:0/1991641713 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fb21402ff10 0x7fb2140323d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:49.420 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.417+0000 7fb2301c6700 1 --2- 172.21.15.184:0/1991641713 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb2280f6a80 0x7fb2281094a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:49.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.417+0000 7fb2301c6700 1 -- 172.21.15.184:0/1991641713 >> 172.21.15.184:0/1991641713 conn(0x7fb2280f02e0 msgr2=0x7fb2280f0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:49.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.417+0000 7fb2301c6700 1 -- 172.21.15.184:0/1991641713 shutdown_connections 2024-02-19T22:19:49.421 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:49.417+0000 7fb2301c6700 1 -- 172.21.15.184:0/1991641713 wait complete. 2024-02-19T22:19:49.421 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:50.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:49 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:48.406004+0000 mgr.smithi176.faoogw (mgr.14162) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:50.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:49 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:49.417130+0000 mon.smithi176 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.184:0/1991641713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:51.131 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:51.132 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:52.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:51 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:50.406285+0000 mgr.smithi176.faoogw (mgr.14162) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:53.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.052+0000 7f3dde3fc700 1 -- 172.21.15.184:0/3892579417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd80f4740 msgr2=0x7f3dd80f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:53.056 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.052+0000 7f3dde3fc700 1 --2- 172.21.15.184:0/3892579417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd80f4740 0x7f3dd80f4b60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f3dcc009fb0 tx=0x7f3dcc009b30).stop 2024-02-19T22:19:53.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.052+0000 7f3dde3fc700 1 -- 172.21.15.184:0/3892579417 shutdown_connections 2024-02-19T22:19:53.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.052+0000 7f3dde3fc700 1 --2- 172.21.15.184:0/3892579417 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd80f4740 0x7f3dd80f4b60 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:53.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.052+0000 7f3dde3fc700 1 -- 172.21.15.184:0/3892579417 >> 172.21.15.184:0/3892579417 conn(0x7f3dd80f0260 msgr2=0x7f3dd80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:53.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.052+0000 7f3dde3fc700 1 -- 172.21.15.184:0/3892579417 shutdown_connections 2024-02-19T22:19:53.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.052+0000 7f3dde3fc700 1 -- 172.21.15.184:0/3892579417 wait complete. 2024-02-19T22:19:53.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.053+0000 7f3dde3fc700 1 Processor -- start 2024-02-19T22:19:53.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.053+0000 7f3dde3fc700 1 -- start start 2024-02-19T22:19:53.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.053+0000 7f3dde3fc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd80f4740 0x7f3dd81067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:53.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.053+0000 7f3dde3fc700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd8106ca0 con 0x7f3dd80f4740 2024-02-19T22:19:53.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.053+0000 7f3dd7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd80f4740 0x7f3dd81067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:53.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.053+0000 7f3dd7fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd80f4740 0x7f3dd81067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:48464/0 (socket says 172.21.15.184:48464) 2024-02-19T22:19:53.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.053+0000 7f3dd7fff700 1 -- 172.21.15.184:0/2663698043 learned_addr learned my addr 172.21.15.184:0/2663698043 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:53.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.054+0000 7f3dd7fff700 1 -- 172.21.15.184:0/2663698043 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc005040 con 0x7f3dd80f4740 2024-02-19T22:19:53.057 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.054+0000 7f3dd7fff700 1 --2- 172.21.15.184:0/2663698043 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd80f4740 0x7f3dd81067b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f3dcc005010 tx=0x7f3dcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:53.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.054+0000 7f3dd4ff9700 1 -- 172.21.15.184:0/2663698043 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dcc009320 con 0x7f3dd80f4740 2024-02-19T22:19:53.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.054+0000 7f3dde3fc700 1 -- 172.21.15.184:0/2663698043 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd8106ea0 con 0x7f3dd80f4740 2024-02-19T22:19:53.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.054+0000 7f3dd4ff9700 1 -- 172.21.15.184:0/2663698043 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3dcc013380 con 0x7f3dd80f4740 2024-02-19T22:19:53.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.054+0000 7f3dd4ff9700 1 -- 172.21.15.184:0/2663698043 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dcc014e50 con 0x7f3dd80f4740 2024-02-19T22:19:53.058 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.054+0000 7f3dde3fc700 1 -- 172.21.15.184:0/2663698043 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd8107330 con 0x7f3dd80f4740 2024-02-19T22:19:53.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.055+0000 7f3dd4ff9700 1 -- 172.21.15.184:0/2663698043 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3dcc0263a0 con 0x7f3dd80f4740 2024-02-19T22:19:53.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.055+0000 7f3dde3fc700 1 -- 172.21.15.184:0/2663698043 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd8045ad0 con 0x7f3dd80f4740 2024-02-19T22:19:53.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.055+0000 7f3dd4ff9700 1 --2- 172.21.15.184:0/2663698043 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3dc002ff10 0x7f3dc00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:53.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.055+0000 7f3dd4ff9700 1 -- 172.21.15.184:0/2663698043 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3dcc041250 con 0x7f3dd80f4740 2024-02-19T22:19:53.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.055+0000 7f3dd77fe700 1 --2- 172.21.15.184:0/2663698043 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3dc002ff10 0x7f3dc00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:53.059 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.055+0000 7f3dd77fe700 1 --2- 172.21.15.184:0/2663698043 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3dc002ff10 0x7f3dc00323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3dc80096a0 tx=0x7f3dc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:53.062 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.060+0000 7f3dd4ff9700 1 -- 172.21.15.184:0/2663698043 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3dcc026e60 con 0x7f3dd80f4740 2024-02-19T22:19:53.268 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.265+0000 7f3dde3fc700 1 -- 172.21.15.184:0/2663698043 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3dd81081e0 con 0x7f3dd80f4740 2024-02-19T22:19:53.268 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.266+0000 7f3dd4ff9700 1 -- 172.21.15.184:0/2663698043 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3dcc012330 con 0x7f3dd80f4740 2024-02-19T22:19:53.269 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:53.269 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:53.270 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.268+0000 7f3dde3fc700 1 -- 172.21.15.184:0/2663698043 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3dc002ff10 msgr2=0x7f3dc00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:53.270 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.268+0000 7f3dde3fc700 1 --2- 172.21.15.184:0/2663698043 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3dc002ff10 0x7f3dc00323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3dc80096a0 tx=0x7f3dc8006b40).stop 2024-02-19T22:19:53.270 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.268+0000 7f3dde3fc700 1 -- 172.21.15.184:0/2663698043 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd80f4740 msgr2=0x7f3dd81067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:53.270 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.268+0000 7f3dde3fc700 1 --2- 172.21.15.184:0/2663698043 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd80f4740 0x7f3dd81067b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f3dcc005010 tx=0x7f3dcc00b040).stop 2024-02-19T22:19:53.270 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.268+0000 7f3dde3fc700 1 -- 172.21.15.184:0/2663698043 shutdown_connections 2024-02-19T22:19:53.271 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.268+0000 7f3dde3fc700 1 --2- 172.21.15.184:0/2663698043 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3dc002ff10 0x7f3dc00323d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:53.271 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.268+0000 7f3dde3fc700 1 --2- 172.21.15.184:0/2663698043 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3dd80f4740 0x7f3dd81067b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:53.271 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.268+0000 7f3dde3fc700 1 -- 172.21.15.184:0/2663698043 >> 172.21.15.184:0/2663698043 conn(0x7f3dd80f0260 msgr2=0x7f3dd80f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:53.271 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.268+0000 7f3dde3fc700 1 -- 172.21.15.184:0/2663698043 shutdown_connections 2024-02-19T22:19:53.271 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:53.268+0000 7f3dde3fc700 1 -- 172.21.15.184:0/2663698043 wait complete. 2024-02-19T22:19:53.271 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:54.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:52.406564+0000 mgr.smithi176.faoogw (mgr.14162) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:54.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:53 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:53.267541+0000 mon.smithi176 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.184:0/2663698043' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:55.096 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:55.117 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:56.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:55 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:54.406880+0000 mgr.smithi176.faoogw (mgr.14162) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:56.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.933+0000 7fc9495b7700 1 -- 172.21.15.184:0/2394145522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9440f47c0 msgr2=0x7fc9440f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:56.941 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.933+0000 7fc9495b7700 1 --2- 172.21.15.184:0/2394145522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9440f47c0 0x7fc9440f4be0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fc938009fb0 tx=0x7fc938009b30).stop 2024-02-19T22:19:56.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.933+0000 7fc9495b7700 1 -- 172.21.15.184:0/2394145522 shutdown_connections 2024-02-19T22:19:56.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.933+0000 7fc9495b7700 1 --2- 172.21.15.184:0/2394145522 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9440f47c0 0x7fc9440f4be0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:56.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.933+0000 7fc9495b7700 1 -- 172.21.15.184:0/2394145522 >> 172.21.15.184:0/2394145522 conn(0x7fc9440f02e0 msgr2=0x7fc9440f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:56.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.933+0000 7fc9495b7700 1 -- 172.21.15.184:0/2394145522 shutdown_connections 2024-02-19T22:19:56.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.933+0000 7fc9495b7700 1 -- 172.21.15.184:0/2394145522 wait complete. 2024-02-19T22:19:56.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.934+0000 7fc9495b7700 1 Processor -- start 2024-02-19T22:19:56.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.934+0000 7fc9495b7700 1 -- start start 2024-02-19T22:19:56.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.934+0000 7fc9495b7700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9440f47c0 0x7fc9441067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:56.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.934+0000 7fc9495b7700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc944106cc0 con 0x7fc9440f47c0 2024-02-19T22:19:56.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.935+0000 7fc942ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9440f47c0 0x7fc9441067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:56.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.935+0000 7fc942ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9440f47c0 0x7fc9441067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:57010/0 (socket says 172.21.15.184:57010) 2024-02-19T22:19:56.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.935+0000 7fc942ffd700 1 -- 172.21.15.184:0/522630426 learned_addr learned my addr 172.21.15.184:0/522630426 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:19:56.942 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.935+0000 7fc942ffd700 1 -- 172.21.15.184:0/522630426 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc938005040 con 0x7fc9440f47c0 2024-02-19T22:19:56.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.935+0000 7fc942ffd700 1 --2- 172.21.15.184:0/522630426 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9440f47c0 0x7fc9441067d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fc938005010 tx=0x7fc93800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:56.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.935+0000 7fc92b7fe700 1 -- 172.21.15.184:0/522630426 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc938009320 con 0x7fc9440f47c0 2024-02-19T22:19:56.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.935+0000 7fc9495b7700 1 -- 172.21.15.184:0/522630426 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc944106ec0 con 0x7fc9440f47c0 2024-02-19T22:19:56.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.935+0000 7fc92b7fe700 1 -- 172.21.15.184:0/522630426 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc938013380 con 0x7fc9440f47c0 2024-02-19T22:19:56.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.935+0000 7fc92b7fe700 1 -- 172.21.15.184:0/522630426 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc938014e50 con 0x7fc9440f47c0 2024-02-19T22:19:56.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.935+0000 7fc9495b7700 1 -- 172.21.15.184:0/522630426 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc944107350 con 0x7fc9440f47c0 2024-02-19T22:19:56.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.936+0000 7fc92b7fe700 1 -- 172.21.15.184:0/522630426 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9380095f0 con 0x7fc9440f47c0 2024-02-19T22:19:56.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.936+0000 7fc9495b7700 1 -- 172.21.15.184:0/522630426 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc944045ad0 con 0x7fc9440f47c0 2024-02-19T22:19:56.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.936+0000 7fc92b7fe700 1 --2- 172.21.15.184:0/522630426 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc92c02ff10 0x7fc92c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:56.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.936+0000 7fc92b7fe700 1 -- 172.21.15.184:0/522630426 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc938040fa0 con 0x7fc9440f47c0 2024-02-19T22:19:56.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.936+0000 7fc9427fc700 1 --2- 172.21.15.184:0/522630426 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc92c02ff10 0x7fc92c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:56.943 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.937+0000 7fc9427fc700 1 --2- 172.21.15.184:0/522630426 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc92c02ff10 0x7fc92c0323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fc9340096a0 tx=0x7fc934006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:56.944 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:56.941+0000 7fc92b7fe700 1 -- 172.21.15.184:0/522630426 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc938014420 con 0x7fc9440f47c0 2024-02-19T22:19:57.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:57.145+0000 7fc9495b7700 1 -- 172.21.15.184:0/522630426 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc944108200 con 0x7fc9440f47c0 2024-02-19T22:19:57.147 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:57.145+0000 7fc92b7fe700 1 -- 172.21.15.184:0/522630426 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc938010230 con 0x7fc9440f47c0 2024-02-19T22:19:57.148 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:19:57.148 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:57.149 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:57.147+0000 7fc9495b7700 1 -- 172.21.15.184:0/522630426 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc92c02ff10 msgr2=0x7fc92c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:57.149 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:57.147+0000 7fc9495b7700 1 --2- 172.21.15.184:0/522630426 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc92c02ff10 0x7fc92c0323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fc9340096a0 tx=0x7fc934006b40).stop 2024-02-19T22:19:57.149 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:57.147+0000 7fc9495b7700 1 -- 172.21.15.184:0/522630426 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9440f47c0 msgr2=0x7fc9441067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:57.149 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:57.147+0000 7fc9495b7700 1 --2- 172.21.15.184:0/522630426 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9440f47c0 0x7fc9441067d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fc938005010 tx=0x7fc93800b040).stop 2024-02-19T22:19:57.149 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:57.147+0000 7fc9495b7700 1 -- 172.21.15.184:0/522630426 shutdown_connections 2024-02-19T22:19:57.150 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:57.147+0000 7fc9495b7700 1 --2- 172.21.15.184:0/522630426 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fc92c02ff10 0x7fc92c0323d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:57.150 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:57.147+0000 7fc9495b7700 1 --2- 172.21.15.184:0/522630426 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc9440f47c0 0x7fc9441067d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:57.150 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:57.147+0000 7fc9495b7700 1 -- 172.21.15.184:0/522630426 >> 172.21.15.184:0/522630426 conn(0x7fc9440f02e0 msgr2=0x7fc9440f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:57.150 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:57.147+0000 7fc9495b7700 1 -- 172.21.15.184:0/522630426 shutdown_connections 2024-02-19T22:19:57.150 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:19:57.147+0000 7fc9495b7700 1 -- 172.21.15.184:0/522630426 wait complete. 2024-02-19T22:19:57.150 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:19:58.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:56.407219+0000 mgr.smithi176.faoogw (mgr.14162) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:58.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:57 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:19:57.146859+0000 mon.smithi176 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.184:0/522630426' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:58.786 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:58.786 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:00.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:19:59 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:19:58.407638+0000 mgr.smithi176.faoogw (mgr.14162) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:00.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.703+0000 7f012cb8a700 1 -- 172.21.15.184:0/2089869632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01280f5410 msgr2=0x7f01280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:00.705 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.703+0000 7f012cb8a700 1 --2- 172.21.15.184:0/2089869632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01280f5410 0x7f01280f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f0110009fb0 tx=0x7f0110009b30).stop 2024-02-19T22:20:00.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.703+0000 7f012cb8a700 1 -- 172.21.15.184:0/2089869632 shutdown_connections 2024-02-19T22:20:00.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.703+0000 7f012cb8a700 1 --2- 172.21.15.184:0/2089869632 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01280f5410 0x7f01280f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:00.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.703+0000 7f012cb8a700 1 -- 172.21.15.184:0/2089869632 >> 172.21.15.184:0/2089869632 conn(0x7f01280f0b70 msgr2=0x7f01280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:00.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.703+0000 7f012cb8a700 1 -- 172.21.15.184:0/2089869632 shutdown_connections 2024-02-19T22:20:00.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.703+0000 7f012cb8a700 1 -- 172.21.15.184:0/2089869632 wait complete. 2024-02-19T22:20:00.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.703+0000 7f012cb8a700 1 Processor -- start 2024-02-19T22:20:00.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.703+0000 7f012cb8a700 1 -- start start 2024-02-19T22:20:00.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.703+0000 7f012cb8a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01280f5410 0x7f01280fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:00.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.703+0000 7f012cb8a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01280fb960 con 0x7f01280f5410 2024-02-19T22:20:00.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.704+0000 7f012659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01280f5410 0x7f01280fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:00.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.704+0000 7f012659c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01280f5410 0x7f01280fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:57034/0 (socket says 172.21.15.184:57034) 2024-02-19T22:20:00.706 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.704+0000 7f012659c700 1 -- 172.21.15.184:0/165189397 learned_addr learned my addr 172.21.15.184:0/165189397 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:20:00.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.704+0000 7f012659c700 1 -- 172.21.15.184:0/165189397 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0110005040 con 0x7f01280f5410 2024-02-19T22:20:00.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.704+0000 7f012659c700 1 --2- 172.21.15.184:0/165189397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01280f5410 0x7f01280fd4e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f0110005010 tx=0x7f011000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:00.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.704+0000 7f011effd700 1 -- 172.21.15.184:0/165189397 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0110009320 con 0x7f01280f5410 2024-02-19T22:20:00.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.704+0000 7f012cb8a700 1 -- 172.21.15.184:0/165189397 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01280fbb00 con 0x7f01280f5410 2024-02-19T22:20:00.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.704+0000 7f011effd700 1 -- 172.21.15.184:0/165189397 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0110013380 con 0x7f01280f5410 2024-02-19T22:20:00.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.704+0000 7f011effd700 1 -- 172.21.15.184:0/165189397 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0110014e50 con 0x7f01280f5410 2024-02-19T22:20:00.707 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.704+0000 7f012cb8a700 1 -- 172.21.15.184:0/165189397 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01280fbf90 con 0x7f01280f5410 2024-02-19T22:20:00.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.705+0000 7f011effd700 1 -- 172.21.15.184:0/165189397 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01100095f0 con 0x7f01280f5410 2024-02-19T22:20:00.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.705+0000 7f012cb8a700 1 -- 172.21.15.184:0/165189397 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0128059100 con 0x7f01280f5410 2024-02-19T22:20:00.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.705+0000 7f011effd700 1 --2- 172.21.15.184:0/165189397 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f011402ff10 0x7f01140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:00.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.705+0000 7f011effd700 1 -- 172.21.15.184:0/165189397 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0110040f80 con 0x7f01280f5410 2024-02-19T22:20:00.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.705+0000 7f0125d9b700 1 --2- 172.21.15.184:0/165189397 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f011402ff10 0x7f01140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:00.708 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.706+0000 7f0125d9b700 1 --2- 172.21.15.184:0/165189397 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f011402ff10 0x7f01140323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f01180096a0 tx=0x7f0118006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:00.711 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.710+0000 7f011effd700 1 -- 172.21.15.184:0/165189397 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0110014420 con 0x7f01280f5410 2024-02-19T22:20:00.917 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.915+0000 7f012cb8a700 1 -- 172.21.15.184:0/165189397 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0128045ad0 con 0x7f01280f5410 2024-02-19T22:20:00.917 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.916+0000 7f011effd700 1 -- 172.21.15.184:0/165189397 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0110010230 con 0x7f01280f5410 2024-02-19T22:20:00.918 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:20:00.918 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:00.919 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.918+0000 7f012cb8a700 1 -- 172.21.15.184:0/165189397 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f011402ff10 msgr2=0x7f01140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:00.919 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.918+0000 7f012cb8a700 1 --2- 172.21.15.184:0/165189397 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f011402ff10 0x7f01140323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f01180096a0 tx=0x7f0118006b40).stop 2024-02-19T22:20:00.920 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.918+0000 7f012cb8a700 1 -- 172.21.15.184:0/165189397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01280f5410 msgr2=0x7f01280fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:00.920 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.918+0000 7f012cb8a700 1 --2- 172.21.15.184:0/165189397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01280f5410 0x7f01280fd4e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f0110005010 tx=0x7f011000b040).stop 2024-02-19T22:20:00.920 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.918+0000 7f012cb8a700 1 -- 172.21.15.184:0/165189397 shutdown_connections 2024-02-19T22:20:00.920 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.918+0000 7f012cb8a700 1 --2- 172.21.15.184:0/165189397 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f011402ff10 0x7f01140323d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:00.920 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.918+0000 7f012cb8a700 1 --2- 172.21.15.184:0/165189397 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f01280f5410 0x7f01280fd4e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:00.920 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.918+0000 7f012cb8a700 1 -- 172.21.15.184:0/165189397 >> 172.21.15.184:0/165189397 conn(0x7f01280f0b70 msgr2=0x7f012805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:00.920 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.918+0000 7f012cb8a700 1 -- 172.21.15.184:0/165189397 shutdown_connections 2024-02-19T22:20:00.920 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:00.918+0000 7f012cb8a700 1 -- 172.21.15.184:0/165189397 wait complete. 2024-02-19T22:20:00.920 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:20:01.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:00 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:00.407991+0000 mgr.smithi176.faoogw (mgr.14162) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:02.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:01 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:20:00.916722+0000 mon.smithi176 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.184:0/165189397' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:02.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:02.680 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:03.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:02 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:02.408375+0000 mgr.smithi176.faoogw (mgr.14162) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:04.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.583+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/4271209261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3a40f4740 msgr2=0x7fa3a40f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:04.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.583+0000 7fa3a8b8a700 1 --2- 172.21.15.184:0/4271209261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3a40f4740 0x7fa3a40f4b60 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fa398009fb0 tx=0x7fa398009b30).stop 2024-02-19T22:20:04.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.583+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/4271209261 shutdown_connections 2024-02-19T22:20:04.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.583+0000 7fa3a8b8a700 1 --2- 172.21.15.184:0/4271209261 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3a40f4740 0x7fa3a40f4b60 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:04.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.583+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/4271209261 >> 172.21.15.184:0/4271209261 conn(0x7fa3a40f0260 msgr2=0x7fa3a40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:04.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.583+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/4271209261 shutdown_connections 2024-02-19T22:20:04.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.583+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/4271209261 wait complete. 2024-02-19T22:20:04.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.583+0000 7fa3a8b8a700 1 Processor -- start 2024-02-19T22:20:04.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.583+0000 7fa3a8b8a700 1 -- start start 2024-02-19T22:20:04.587 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.584+0000 7fa3a8b8a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3a40f4740 0x7fa3a41067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:04.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.584+0000 7fa3a8b8a700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a4106ca0 con 0x7fa3a40f4740 2024-02-19T22:20:04.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.584+0000 7fa3a259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3a40f4740 0x7fa3a41067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:04.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.584+0000 7fa3a259c700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3a40f4740 0x7fa3a41067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:57058/0 (socket says 172.21.15.184:57058) 2024-02-19T22:20:04.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.584+0000 7fa3a259c700 1 -- 172.21.15.184:0/1713463206 learned_addr learned my addr 172.21.15.184:0/1713463206 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:20:04.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.584+0000 7fa3a259c700 1 -- 172.21.15.184:0/1713463206 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa398005040 con 0x7fa3a40f4740 2024-02-19T22:20:04.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.584+0000 7fa3a259c700 1 --2- 172.21.15.184:0/1713463206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3a40f4740 0x7fa3a41067b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fa398005010 tx=0x7fa39800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:04.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.584+0000 7fa392ffd700 1 -- 172.21.15.184:0/1713463206 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa398009320 con 0x7fa3a40f4740 2024-02-19T22:20:04.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.584+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/1713463206 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a4106ea0 con 0x7fa3a40f4740 2024-02-19T22:20:04.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.584+0000 7fa392ffd700 1 -- 172.21.15.184:0/1713463206 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa398013380 con 0x7fa3a40f4740 2024-02-19T22:20:04.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.584+0000 7fa392ffd700 1 -- 172.21.15.184:0/1713463206 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa398014e50 con 0x7fa3a40f4740 2024-02-19T22:20:04.588 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.585+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/1713463206 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a4107330 con 0x7fa3a40f4740 2024-02-19T22:20:04.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.585+0000 7fa392ffd700 1 -- 172.21.15.184:0/1713463206 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3980095f0 con 0x7fa3a40f4740 2024-02-19T22:20:04.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.585+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/1713463206 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a4045ad0 con 0x7fa3a40f4740 2024-02-19T22:20:04.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.585+0000 7fa392ffd700 1 --2- 172.21.15.184:0/1713463206 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa38c02ff10 0x7fa38c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:04.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.585+0000 7fa392ffd700 1 -- 172.21.15.184:0/1713463206 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa398040fa0 con 0x7fa3a40f4740 2024-02-19T22:20:04.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.586+0000 7fa3a1d9b700 1 --2- 172.21.15.184:0/1713463206 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa38c02ff10 0x7fa38c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:04.589 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.586+0000 7fa3a1d9b700 1 --2- 172.21.15.184:0/1713463206 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa38c02ff10 0x7fa38c0323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fa3940096a0 tx=0x7fa394006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:04.592 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.590+0000 7fa392ffd700 1 -- 172.21.15.184:0/1713463206 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa398014420 con 0x7fa3a40f4740 2024-02-19T22:20:04.798 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.796+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/1713463206 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3a41081e0 con 0x7fa3a40f4740 2024-02-19T22:20:04.799 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.797+0000 7fa392ffd700 1 -- 172.21.15.184:0/1713463206 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa398010230 con 0x7fa3a40f4740 2024-02-19T22:20:04.799 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:20:04.799 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:04.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.799+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/1713463206 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa38c02ff10 msgr2=0x7fa38c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:04.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.799+0000 7fa3a8b8a700 1 --2- 172.21.15.184:0/1713463206 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa38c02ff10 0x7fa38c0323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fa3940096a0 tx=0x7fa394006b40).stop 2024-02-19T22:20:04.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.799+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/1713463206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3a40f4740 msgr2=0x7fa3a41067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:04.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.799+0000 7fa3a8b8a700 1 --2- 172.21.15.184:0/1713463206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3a40f4740 0x7fa3a41067b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fa398005010 tx=0x7fa39800b040).stop 2024-02-19T22:20:04.801 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.799+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/1713463206 shutdown_connections 2024-02-19T22:20:04.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.799+0000 7fa3a8b8a700 1 --2- 172.21.15.184:0/1713463206 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa38c02ff10 0x7fa38c0323d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:04.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.799+0000 7fa3a8b8a700 1 --2- 172.21.15.184:0/1713463206 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa3a40f4740 0x7fa3a41067b0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:04.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.799+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/1713463206 >> 172.21.15.184:0/1713463206 conn(0x7fa3a40f0260 msgr2=0x7fa3a40f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:04.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.799+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/1713463206 shutdown_connections 2024-02-19T22:20:04.802 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:04.799+0000 7fa3a8b8a700 1 -- 172.21.15.184:0/1713463206 wait complete. 2024-02-19T22:20:04.802 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:20:05.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:04.408655+0000 mgr.smithi176.faoogw (mgr.14162) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:05.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:05 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:20:04.798631+0000 mon.smithi176 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.184:0/1713463206' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:06.524 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:06.525 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:07.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:07 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:06.409034+0000 mgr.smithi176.faoogw (mgr.14162) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.600+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/1243193744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cf40f6510 msgr2=0x7f3cf40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.600+0000 7f3cfb0d8700 1 --2- 172.21.15.184:0/1243193744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cf40f6510 0x7f3cf40f6930 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f3ce8009fb0 tx=0x7f3ce8009b30).stop 2024-02-19T22:20:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.600+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/1243193744 shutdown_connections 2024-02-19T22:20:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.600+0000 7f3cfb0d8700 1 --2- 172.21.15.184:0/1243193744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cf40f6510 0x7f3cf40f6930 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.600+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/1243193744 >> 172.21.15.184:0/1243193744 conn(0x7f3cf40f1c70 msgr2=0x7f3cf40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.600+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/1243193744 shutdown_connections 2024-02-19T22:20:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.600+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/1243193744 wait complete. 2024-02-19T22:20:08.603 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.601+0000 7f3cfb0d8700 1 Processor -- start 2024-02-19T22:20:08.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.601+0000 7f3cfb0d8700 1 -- start start 2024-02-19T22:20:08.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.601+0000 7f3cfb0d8700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cf40f6510 0x7f3cf41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:08.604 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.601+0000 7f3cfb0d8700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4106cd0 con 0x7f3cf40f6510 2024-02-19T22:20:08.605 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.601+0000 7f3cf8e74700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cf40f6510 0x7f3cf41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:08.605 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.601+0000 7f3cf8e74700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cf40f6510 0x7f3cf41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:40924/0 (socket says 172.21.15.184:40924) 2024-02-19T22:20:08.605 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.601+0000 7f3cf8e74700 1 -- 172.21.15.184:0/4192613337 learned_addr learned my addr 172.21.15.184:0/4192613337 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:20:08.605 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.602+0000 7f3cf8e74700 1 -- 172.21.15.184:0/4192613337 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce8005040 con 0x7f3cf40f6510 2024-02-19T22:20:08.605 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.602+0000 7f3cf8e74700 1 --2- 172.21.15.184:0/4192613337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cf40f6510 0x7f3cf41067e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f3ce8005010 tx=0x7f3ce800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:08.605 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.602+0000 7f3cf17fa700 1 -- 172.21.15.184:0/4192613337 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce8009320 con 0x7f3cf40f6510 2024-02-19T22:20:08.606 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.602+0000 7f3cf17fa700 1 -- 172.21.15.184:0/4192613337 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ce8013380 con 0x7f3cf40f6510 2024-02-19T22:20:08.606 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.602+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/4192613337 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf4106ed0 con 0x7f3cf40f6510 2024-02-19T22:20:08.606 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.602+0000 7f3cf17fa700 1 -- 172.21.15.184:0/4192613337 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce8014e50 con 0x7f3cf40f6510 2024-02-19T22:20:08.606 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.602+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/4192613337 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf4107360 con 0x7f3cf40f6510 2024-02-19T22:20:08.606 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.603+0000 7f3cf17fa700 1 -- 172.21.15.184:0/4192613337 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ce8014420 con 0x7f3cf40f6510 2024-02-19T22:20:08.606 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.603+0000 7f3cf17fa700 1 --2- 172.21.15.184:0/4192613337 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3ce402ff10 0x7f3ce40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:08.606 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.603+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/4192613337 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf4045ad0 con 0x7f3cf40f6510 2024-02-19T22:20:08.606 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.603+0000 7f3cf17fa700 1 -- 172.21.15.184:0/4192613337 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ce8040f40 con 0x7f3cf40f6510 2024-02-19T22:20:08.606 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.603+0000 7f3cf3fff700 1 --2- 172.21.15.184:0/4192613337 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3ce402ff10 0x7f3ce40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:08.606 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.604+0000 7f3cf3fff700 1 --2- 172.21.15.184:0/4192613337 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3ce402ff10 0x7f3ce40323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f3ce00096a0 tx=0x7f3ce0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:08.610 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.608+0000 7f3cf17fa700 1 -- 172.21.15.184:0/4192613337 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ce801db70 con 0x7f3cf40f6510 2024-02-19T22:20:08.819 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.817+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/4192613337 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cf4108210 con 0x7f3cf40f6510 2024-02-19T22:20:08.820 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.818+0000 7f3cf17fa700 1 -- 172.21.15.184:0/4192613337 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ce801db70 con 0x7f3cf40f6510 2024-02-19T22:20:08.820 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:20:08.820 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:08.822 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.820+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/4192613337 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3ce402ff10 msgr2=0x7f3ce40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:08.822 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.820+0000 7f3cfb0d8700 1 --2- 172.21.15.184:0/4192613337 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3ce402ff10 0x7f3ce40323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f3ce00096a0 tx=0x7f3ce0006b40).stop 2024-02-19T22:20:08.822 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.820+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/4192613337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cf40f6510 msgr2=0x7f3cf41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:08.822 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.820+0000 7f3cfb0d8700 1 --2- 172.21.15.184:0/4192613337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cf40f6510 0x7f3cf41067e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f3ce8005010 tx=0x7f3ce800b040).stop 2024-02-19T22:20:08.822 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.820+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/4192613337 shutdown_connections 2024-02-19T22:20:08.822 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.820+0000 7f3cfb0d8700 1 --2- 172.21.15.184:0/4192613337 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f3ce402ff10 0x7f3ce40323d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:08.823 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.820+0000 7f3cfb0d8700 1 --2- 172.21.15.184:0/4192613337 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3cf40f6510 0x7f3cf41067e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:08.823 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.820+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/4192613337 >> 172.21.15.184:0/4192613337 conn(0x7f3cf40f1c70 msgr2=0x7f3cf40f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:08.823 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.820+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/4192613337 shutdown_connections 2024-02-19T22:20:08.823 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:08.820+0000 7f3cfb0d8700 1 -- 172.21.15.184:0/4192613337 wait complete. 2024-02-19T22:20:08.823 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:20:09.885 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:08.409353+0000 mgr.smithi176.faoogw (mgr.14162) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:09.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:09 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:20:08.819422+0000 mon.smithi176 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.184:0/4192613337' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:10.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:10.497 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:12.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:11 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:10.409680+0000 mgr.smithi176.faoogw (mgr.14162) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:12.349 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.345+0000 7ff4d21bf700 1 -- 172.21.15.184:0/2690649509 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4cc0f6510 msgr2=0x7ff4cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:12.349 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.345+0000 7ff4d21bf700 1 --2- 172.21.15.184:0/2690649509 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4cc0f6510 0x7ff4cc0f6930 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7ff4c0009fb0 tx=0x7ff4c0009b30).stop 2024-02-19T22:20:12.349 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.345+0000 7ff4d21bf700 1 -- 172.21.15.184:0/2690649509 shutdown_connections 2024-02-19T22:20:12.349 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.345+0000 7ff4d21bf700 1 --2- 172.21.15.184:0/2690649509 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4cc0f6510 0x7ff4cc0f6930 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:12.349 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.345+0000 7ff4d21bf700 1 -- 172.21.15.184:0/2690649509 >> 172.21.15.184:0/2690649509 conn(0x7ff4cc0f1c70 msgr2=0x7ff4cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:12.349 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.345+0000 7ff4d21bf700 1 -- 172.21.15.184:0/2690649509 shutdown_connections 2024-02-19T22:20:12.349 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.345+0000 7ff4d21bf700 1 -- 172.21.15.184:0/2690649509 wait complete. 2024-02-19T22:20:12.349 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.346+0000 7ff4d21bf700 1 Processor -- start 2024-02-19T22:20:12.349 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.346+0000 7ff4d21bf700 1 -- start start 2024-02-19T22:20:12.349 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.346+0000 7ff4d21bf700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4cc0f6510 0x7ff4cc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:12.350 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.346+0000 7ff4d21bf700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc069e20 con 0x7ff4cc0f6510 2024-02-19T22:20:12.350 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.346+0000 7ff4cb7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4cc0f6510 0x7ff4cc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:12.350 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.346+0000 7ff4cb7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4cc0f6510 0x7ff4cc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:40942/0 (socket says 172.21.15.184:40942) 2024-02-19T22:20:12.350 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.346+0000 7ff4cb7fe700 1 -- 172.21.15.184:0/1825778607 learned_addr learned my addr 172.21.15.184:0/1825778607 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:20:12.350 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.346+0000 7ff4cb7fe700 1 -- 172.21.15.184:0/1825778607 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c0005040 con 0x7ff4cc0f6510 2024-02-19T22:20:12.350 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.347+0000 7ff4cb7fe700 1 --2- 172.21.15.184:0/1825778607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4cc0f6510 0x7ff4cc06b9a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7ff4c0005010 tx=0x7ff4c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:12.350 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.347+0000 7ff4b3fff700 1 -- 172.21.15.184:0/1825778607 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4c0009320 con 0x7ff4cc0f6510 2024-02-19T22:20:12.350 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.347+0000 7ff4d21bf700 1 -- 172.21.15.184:0/1825778607 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4cc069fc0 con 0x7ff4cc0f6510 2024-02-19T22:20:12.350 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.347+0000 7ff4b3fff700 1 -- 172.21.15.184:0/1825778607 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff4c0013380 con 0x7ff4cc0f6510 2024-02-19T22:20:12.350 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.347+0000 7ff4b3fff700 1 -- 172.21.15.184:0/1825778607 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4c0014e50 con 0x7ff4cc0f6510 2024-02-19T22:20:12.350 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.347+0000 7ff4d21bf700 1 -- 172.21.15.184:0/1825778607 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4cc06a450 con 0x7ff4cc0f6510 2024-02-19T22:20:12.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.348+0000 7ff4b3fff700 1 -- 172.21.15.184:0/1825778607 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4c0014420 con 0x7ff4cc0f6510 2024-02-19T22:20:12.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.348+0000 7ff4d21bf700 1 -- 172.21.15.184:0/1825778607 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4cc059100 con 0x7ff4cc0f6510 2024-02-19T22:20:12.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.348+0000 7ff4b3fff700 1 --2- 172.21.15.184:0/1825778607 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff4b402ff10 0x7ff4b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:12.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.348+0000 7ff4b3fff700 1 -- 172.21.15.184:0/1825778607 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff4c0040e70 con 0x7ff4cc0f6510 2024-02-19T22:20:12.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.348+0000 7ff4caffd700 1 --2- 172.21.15.184:0/1825778607 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff4b402ff10 0x7ff4b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:12.351 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.348+0000 7ff4caffd700 1 --2- 172.21.15.184:0/1825778607 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff4b402ff10 0x7ff4b40323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff4bc0096a0 tx=0x7ff4bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:12.354 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.352+0000 7ff4b3fff700 1 -- 172.21.15.184:0/1825778607 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff4c00095f0 con 0x7ff4cc0f6510 2024-02-19T22:20:12.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.557+0000 7ff4d21bf700 1 -- 172.21.15.184:0/1825778607 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4cc0fc0a0 con 0x7ff4cc0f6510 2024-02-19T22:20:12.559 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.557+0000 7ff4b3fff700 1 -- 172.21.15.184:0/1825778607 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff4c001db70 con 0x7ff4cc0f6510 2024-02-19T22:20:12.560 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:20:12.560 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:12.561 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.559+0000 7ff4d21bf700 1 -- 172.21.15.184:0/1825778607 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff4b402ff10 msgr2=0x7ff4b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:12.561 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.559+0000 7ff4d21bf700 1 --2- 172.21.15.184:0/1825778607 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff4b402ff10 0x7ff4b40323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff4bc0096a0 tx=0x7ff4bc006b40).stop 2024-02-19T22:20:12.561 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.559+0000 7ff4d21bf700 1 -- 172.21.15.184:0/1825778607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4cc0f6510 msgr2=0x7ff4cc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:12.561 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.559+0000 7ff4d21bf700 1 --2- 172.21.15.184:0/1825778607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4cc0f6510 0x7ff4cc06b9a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7ff4c0005010 tx=0x7ff4c000b040).stop 2024-02-19T22:20:12.561 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.559+0000 7ff4d21bf700 1 -- 172.21.15.184:0/1825778607 shutdown_connections 2024-02-19T22:20:12.561 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.559+0000 7ff4d21bf700 1 --2- 172.21.15.184:0/1825778607 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7ff4b402ff10 0x7ff4b40323d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:12.561 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.559+0000 7ff4d21bf700 1 --2- 172.21.15.184:0/1825778607 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff4cc0f6510 0x7ff4cc06b9a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:12.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.559+0000 7ff4d21bf700 1 -- 172.21.15.184:0/1825778607 >> 172.21.15.184:0/1825778607 conn(0x7ff4cc0f1c70 msgr2=0x7ff4cc0f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:12.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.559+0000 7ff4d21bf700 1 -- 172.21.15.184:0/1825778607 shutdown_connections 2024-02-19T22:20:12.562 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:12.559+0000 7ff4d21bf700 1 -- 172.21.15.184:0/1825778607 wait complete. 2024-02-19T22:20:12.562 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:20:13.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:12 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:20:12.558777+0000 mon.smithi176 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.184:0/1825778607' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:14.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:13 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:12.410025+0000 mgr.smithi176.faoogw (mgr.14162) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:14.238 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:14.239 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:16.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.076+0000 7f8b950ea700 1 -- 172.21.15.184:0/2361969100 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b900f53f0 msgr2=0x7f8b900f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:16.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.076+0000 7f8b950ea700 1 --2- 172.21.15.184:0/2361969100 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b900f53f0 0x7f8b900f5810 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f8b84009fb0 tx=0x7f8b84009b30).stop 2024-02-19T22:20:16.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.076+0000 7f8b950ea700 1 -- 172.21.15.184:0/2361969100 shutdown_connections 2024-02-19T22:20:16.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.076+0000 7f8b950ea700 1 --2- 172.21.15.184:0/2361969100 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b900f53f0 0x7f8b900f5810 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:16.080 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.076+0000 7f8b950ea700 1 -- 172.21.15.184:0/2361969100 >> 172.21.15.184:0/2361969100 conn(0x7f8b900f0b70 msgr2=0x7f8b900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:16.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.076+0000 7f8b950ea700 1 -- 172.21.15.184:0/2361969100 shutdown_connections 2024-02-19T22:20:16.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.076+0000 7f8b950ea700 1 -- 172.21.15.184:0/2361969100 wait complete. 2024-02-19T22:20:16.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.077+0000 7f8b950ea700 1 Processor -- start 2024-02-19T22:20:16.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.077+0000 7f8b950ea700 1 -- start start 2024-02-19T22:20:16.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.077+0000 7f8b950ea700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b900f53f0 0x7f8b901094c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:16.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.077+0000 7f8b950ea700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b90060240 con 0x7f8b900f53f0 2024-02-19T22:20:16.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.077+0000 7f8b8ed9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b900f53f0 0x7f8b901094c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:16.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.077+0000 7f8b8ed9d700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b900f53f0 0x7f8b901094c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:40956/0 (socket says 172.21.15.184:40956) 2024-02-19T22:20:16.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.077+0000 7f8b8ed9d700 1 -- 172.21.15.184:0/2283428478 learned_addr learned my addr 172.21.15.184:0/2283428478 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:20:16.081 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.077+0000 7f8b8ed9d700 1 -- 172.21.15.184:0/2283428478 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b84005040 con 0x7f8b900f53f0 2024-02-19T22:20:16.082 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.077+0000 7f8b8ed9d700 1 --2- 172.21.15.184:0/2283428478 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b900f53f0 0x7f8b901094c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f8b84005010 tx=0x7f8b8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:16.082 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.078+0000 7f8b777fe700 1 -- 172.21.15.184:0/2283428478 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b84009320 con 0x7f8b900f53f0 2024-02-19T22:20:16.082 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.078+0000 7f8b950ea700 1 -- 172.21.15.184:0/2283428478 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b90060440 con 0x7f8b900f53f0 2024-02-19T22:20:16.082 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.078+0000 7f8b777fe700 1 -- 172.21.15.184:0/2283428478 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b84013380 con 0x7f8b900f53f0 2024-02-19T22:20:16.082 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.078+0000 7f8b777fe700 1 -- 172.21.15.184:0/2283428478 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b84014e50 con 0x7f8b900f53f0 2024-02-19T22:20:16.082 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.078+0000 7f8b950ea700 1 -- 172.21.15.184:0/2283428478 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b90109b70 con 0x7f8b900f53f0 2024-02-19T22:20:16.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.078+0000 7f8b950ea700 1 -- 172.21.15.184:0/2283428478 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b90045ad0 con 0x7f8b900f53f0 2024-02-19T22:20:16.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.079+0000 7f8b777fe700 1 -- 172.21.15.184:0/2283428478 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b840095f0 con 0x7f8b900f53f0 2024-02-19T22:20:16.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.079+0000 7f8b777fe700 1 --2- 172.21.15.184:0/2283428478 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8b7802ff10 0x7f8b780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:16.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.079+0000 7f8b777fe700 1 -- 172.21.15.184:0/2283428478 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b84040e90 con 0x7f8b900f53f0 2024-02-19T22:20:16.083 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.079+0000 7f8b8e59c700 1 --2- 172.21.15.184:0/2283428478 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8b7802ff10 0x7f8b780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:16.084 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.079+0000 7f8b8e59c700 1 --2- 172.21.15.184:0/2283428478 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8b7802ff10 0x7f8b780323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f8b800096a0 tx=0x7f8b80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:16.085 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.083+0000 7f8b777fe700 1 -- 172.21.15.184:0/2283428478 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b84014420 con 0x7f8b900f53f0 2024-02-19T22:20:16.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:15 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:14.410403+0000 mgr.smithi176.faoogw (mgr.14162) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:16.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.288+0000 7f8b950ea700 1 -- 172.21.15.184:0/2283428478 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b9010a390 con 0x7f8b900f53f0 2024-02-19T22:20:16.291 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.289+0000 7f8b777fe700 1 -- 172.21.15.184:0/2283428478 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8b84010230 con 0x7f8b900f53f0 2024-02-19T22:20:16.291 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:20:16.291 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:16.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.291+0000 7f8b950ea700 1 -- 172.21.15.184:0/2283428478 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8b7802ff10 msgr2=0x7f8b780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:16.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.291+0000 7f8b950ea700 1 --2- 172.21.15.184:0/2283428478 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8b7802ff10 0x7f8b780323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f8b800096a0 tx=0x7f8b80006b40).stop 2024-02-19T22:20:16.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.291+0000 7f8b950ea700 1 -- 172.21.15.184:0/2283428478 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b900f53f0 msgr2=0x7f8b901094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:16.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.291+0000 7f8b950ea700 1 --2- 172.21.15.184:0/2283428478 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b900f53f0 0x7f8b901094c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f8b84005010 tx=0x7f8b8400b040).stop 2024-02-19T22:20:16.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.291+0000 7f8b950ea700 1 -- 172.21.15.184:0/2283428478 shutdown_connections 2024-02-19T22:20:16.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.291+0000 7f8b950ea700 1 --2- 172.21.15.184:0/2283428478 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f8b7802ff10 0x7f8b780323d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:16.293 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.291+0000 7f8b950ea700 1 --2- 172.21.15.184:0/2283428478 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8b900f53f0 0x7f8b901094c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:16.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.291+0000 7f8b950ea700 1 -- 172.21.15.184:0/2283428478 >> 172.21.15.184:0/2283428478 conn(0x7f8b900f0b70 msgr2=0x7f8b900f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:16.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.291+0000 7f8b950ea700 1 -- 172.21.15.184:0/2283428478 shutdown_connections 2024-02-19T22:20:16.294 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:16.291+0000 7f8b950ea700 1 -- 172.21.15.184:0/2283428478 wait complete. 2024-02-19T22:20:16.294 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:20:17.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:16 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:20:16.290578+0000 mon.smithi176 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.184:0/2283428478' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:17.996 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:17.996 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:18.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:17 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:16.410780+0000 mgr.smithi176.faoogw (mgr.14162) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:19.806 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.802+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3910250276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6edc0f5410 msgr2=0x7f6edc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:19.806 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.802+0000 7f6ee3e9e700 1 --2- 172.21.15.184:0/3910250276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6edc0f5410 0x7f6edc0f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f6ecc009fb0 tx=0x7f6ecc009b30).stop 2024-02-19T22:20:19.806 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.802+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3910250276 shutdown_connections 2024-02-19T22:20:19.806 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.802+0000 7f6ee3e9e700 1 --2- 172.21.15.184:0/3910250276 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6edc0f5410 0x7f6edc0f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:19.806 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.802+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3910250276 >> 172.21.15.184:0/3910250276 conn(0x7f6edc0f0b70 msgr2=0x7f6edc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.803+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3910250276 shutdown_connections 2024-02-19T22:20:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.803+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3910250276 wait complete. 2024-02-19T22:20:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.803+0000 7f6ee3e9e700 1 Processor -- start 2024-02-19T22:20:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.803+0000 7f6ee3e9e700 1 -- start start 2024-02-19T22:20:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.803+0000 7f6ee3e9e700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6edc0f5410 0x7f6edc1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.803+0000 7f6ee3e9e700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc060240 con 0x7f6edc0f5410 2024-02-19T22:20:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.803+0000 7f6ee1c3a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6edc0f5410 0x7f6edc1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.803+0000 7f6ee1c3a700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6edc0f5410 0x7f6edc1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:59252/0 (socket says 172.21.15.184:59252) 2024-02-19T22:20:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.803+0000 7f6ee1c3a700 1 -- 172.21.15.184:0/3517097863 learned_addr learned my addr 172.21.15.184:0/3517097863 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:20:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.804+0000 7f6ee1c3a700 1 -- 172.21.15.184:0/3517097863 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc005040 con 0x7f6edc0f5410 2024-02-19T22:20:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.804+0000 7f6ee1c3a700 1 --2- 172.21.15.184:0/3517097863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6edc0f5410 0x7f6edc1094e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f6ecc005010 tx=0x7f6ecc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.804+0000 7f6ed27fc700 1 -- 172.21.15.184:0/3517097863 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ecc009320 con 0x7f6edc0f5410 2024-02-19T22:20:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.804+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3517097863 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6edc060440 con 0x7f6edc0f5410 2024-02-19T22:20:19.807 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.804+0000 7f6ed27fc700 1 -- 172.21.15.184:0/3517097863 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ecc013380 con 0x7f6edc0f5410 2024-02-19T22:20:19.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.804+0000 7f6ed27fc700 1 -- 172.21.15.184:0/3517097863 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ecc014e50 con 0x7f6edc0f5410 2024-02-19T22:20:19.808 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.804+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3517097863 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6edc109b90 con 0x7f6edc0f5410 2024-02-19T22:20:19.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.805+0000 7f6ed27fc700 1 -- 172.21.15.184:0/3517097863 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ecc014420 con 0x7f6edc0f5410 2024-02-19T22:20:19.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.805+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3517097863 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6edc045ad0 con 0x7f6edc0f5410 2024-02-19T22:20:19.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.805+0000 7f6ed27fc700 1 --2- 172.21.15.184:0/3517097863 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f6ec802ff10 0x7f6ec80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:19.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.805+0000 7f6ed27fc700 1 -- 172.21.15.184:0/3517097863 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ecc040f00 con 0x7f6edc0f5410 2024-02-19T22:20:19.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.805+0000 7f6ee1439700 1 --2- 172.21.15.184:0/3517097863 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f6ec802ff10 0x7f6ec80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:19.809 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.806+0000 7f6ee1439700 1 --2- 172.21.15.184:0/3517097863 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f6ec802ff10 0x7f6ec80323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f6ed80096a0 tx=0x7f6ed8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:19.811 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:19.809+0000 7f6ed27fc700 1 -- 172.21.15.184:0/3517097863 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ecc0095f0 con 0x7f6edc0f5410 2024-02-19T22:20:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:20.012+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3517097863 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6edc10a3b0 con 0x7f6edc0f5410 2024-02-19T22:20:20.015 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:20.013+0000 7f6ed27fc700 1 -- 172.21.15.184:0/3517097863 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6ecc01db70 con 0x7f6edc0f5410 2024-02-19T22:20:20.015 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:20:20.015 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:20.015+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3517097863 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f6ec802ff10 msgr2=0x7f6ec80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:20.015+0000 7f6ee3e9e700 1 --2- 172.21.15.184:0/3517097863 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f6ec802ff10 0x7f6ec80323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f6ed80096a0 tx=0x7f6ed8006b40).stop 2024-02-19T22:20:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:20.015+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3517097863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6edc0f5410 msgr2=0x7f6edc1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:20.015+0000 7f6ee3e9e700 1 --2- 172.21.15.184:0/3517097863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6edc0f5410 0x7f6edc1094e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f6ecc005010 tx=0x7f6ecc00b040).stop 2024-02-19T22:20:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:20.015+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3517097863 shutdown_connections 2024-02-19T22:20:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:20.015+0000 7f6ee3e9e700 1 --2- 172.21.15.184:0/3517097863 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f6ec802ff10 0x7f6ec80323d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:20.015+0000 7f6ee3e9e700 1 --2- 172.21.15.184:0/3517097863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6edc0f5410 0x7f6edc1094e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:20.015+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3517097863 >> 172.21.15.184:0/3517097863 conn(0x7f6edc0f0b70 msgr2=0x7f6edc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:20.017 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:20.015+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3517097863 shutdown_connections 2024-02-19T22:20:20.018 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:20.015+0000 7f6ee3e9e700 1 -- 172.21.15.184:0/3517097863 wait complete. 2024-02-19T22:20:20.018 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:20:20.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:19 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:18.411162+0000 mgr.smithi176.faoogw (mgr.14162) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:21.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:20 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:20:20.014600+0000 mon.smithi176 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.184:0/3517097863' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:21.669 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:21.670 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:22.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:21 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:20.411469+0000 mgr.smithi176.faoogw (mgr.14162) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:23.514 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.510+0000 7f4223755700 1 -- 172.21.15.184:0/477467934 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0f5410 msgr2=0x7f421c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:23.514 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.510+0000 7f4223755700 1 --2- 172.21.15.184:0/477467934 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0f5410 0x7f421c0f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f420c009fb0 tx=0x7f420c009b30).stop 2024-02-19T22:20:23.514 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.510+0000 7f4223755700 1 -- 172.21.15.184:0/477467934 shutdown_connections 2024-02-19T22:20:23.514 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.510+0000 7f4223755700 1 --2- 172.21.15.184:0/477467934 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0f5410 0x7f421c0f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:23.514 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.510+0000 7f4223755700 1 -- 172.21.15.184:0/477467934 >> 172.21.15.184:0/477467934 conn(0x7f421c0f0b70 msgr2=0x7f421c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:23.514 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.510+0000 7f4223755700 1 -- 172.21.15.184:0/477467934 shutdown_connections 2024-02-19T22:20:23.514 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.510+0000 7f4223755700 1 -- 172.21.15.184:0/477467934 wait complete. 2024-02-19T22:20:23.514 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.511+0000 7f4223755700 1 Processor -- start 2024-02-19T22:20:23.514 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.511+0000 7f4223755700 1 -- start start 2024-02-19T22:20:23.514 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.511+0000 7f4223755700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0f5410 0x7f421c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:23.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.511+0000 7f4223755700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421c060240 con 0x7f421c0f5410 2024-02-19T22:20:23.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.511+0000 7f42214f1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0f5410 0x7f421c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:23.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.511+0000 7f42214f1700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0f5410 0x7f421c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:59270/0 (socket says 172.21.15.184:59270) 2024-02-19T22:20:23.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.511+0000 7f42214f1700 1 -- 172.21.15.184:0/1281309469 learned_addr learned my addr 172.21.15.184:0/1281309469 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:20:23.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.511+0000 7f42214f1700 1 -- 172.21.15.184:0/1281309469 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c005040 con 0x7f421c0f5410 2024-02-19T22:20:23.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.511+0000 7f42214f1700 1 --2- 172.21.15.184:0/1281309469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0f5410 0x7f421c1094e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f420c0089f0 tx=0x7f420c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:23.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.511+0000 7f4211ffb700 1 -- 172.21.15.184:0/1281309469 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f420c009320 con 0x7f421c0f5410 2024-02-19T22:20:23.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.512+0000 7f4223755700 1 -- 172.21.15.184:0/1281309469 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421c060440 con 0x7f421c0f5410 2024-02-19T22:20:23.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.512+0000 7f4211ffb700 1 -- 172.21.15.184:0/1281309469 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f420c013380 con 0x7f421c0f5410 2024-02-19T22:20:23.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.512+0000 7f4211ffb700 1 -- 172.21.15.184:0/1281309469 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f420c01ed80 con 0x7f421c0f5410 2024-02-19T22:20:23.515 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.512+0000 7f4223755700 1 -- 172.21.15.184:0/1281309469 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421c109b90 con 0x7f421c0f5410 2024-02-19T22:20:23.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.512+0000 7f4211ffb700 1 -- 172.21.15.184:0/1281309469 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f420c01e420 con 0x7f421c0f5410 2024-02-19T22:20:23.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.513+0000 7f4223755700 1 -- 172.21.15.184:0/1281309469 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421c045ad0 con 0x7f421c0f5410 2024-02-19T22:20:23.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.513+0000 7f4211ffb700 1 --2- 172.21.15.184:0/1281309469 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f420802ff60 0x7f4208032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:23.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.513+0000 7f4211ffb700 1 -- 172.21.15.184:0/1281309469 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f420c041090 con 0x7f421c0f5410 2024-02-19T22:20:23.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.513+0000 7f4220cf0700 1 --2- 172.21.15.184:0/1281309469 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f420802ff60 0x7f4208032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:23.516 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.513+0000 7f4220cf0700 1 --2- 172.21.15.184:0/1281309469 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f420802ff60 0x7f4208032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f42180096a0 tx=0x7f4218006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:23.519 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.517+0000 7f4211ffb700 1 -- 172.21.15.184:0/1281309469 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f420c0095f0 con 0x7f421c0f5410 2024-02-19T22:20:23.724 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.722+0000 7f4223755700 1 -- 172.21.15.184:0/1281309469 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f421c10a3b0 con 0x7f421c0f5410 2024-02-19T22:20:23.725 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.723+0000 7f4211ffb700 1 -- 172.21.15.184:0/1281309469 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f420c004490 con 0x7f421c0f5410 2024-02-19T22:20:23.725 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:20:23.725 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:23.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.724+0000 7f4223755700 1 -- 172.21.15.184:0/1281309469 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f420802ff60 msgr2=0x7f4208032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:23.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.724+0000 7f4223755700 1 --2- 172.21.15.184:0/1281309469 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f420802ff60 0x7f4208032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f42180096a0 tx=0x7f4218006b40).stop 2024-02-19T22:20:23.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.724+0000 7f4223755700 1 -- 172.21.15.184:0/1281309469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0f5410 msgr2=0x7f421c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:23.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.724+0000 7f4223755700 1 --2- 172.21.15.184:0/1281309469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0f5410 0x7f421c1094e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f420c0089f0 tx=0x7f420c00b040).stop 2024-02-19T22:20:23.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.724+0000 7f4223755700 1 -- 172.21.15.184:0/1281309469 shutdown_connections 2024-02-19T22:20:23.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.724+0000 7f4223755700 1 --2- 172.21.15.184:0/1281309469 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f420802ff60 0x7f4208032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:23.727 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.724+0000 7f4223755700 1 --2- 172.21.15.184:0/1281309469 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f421c0f5410 0x7f421c1094e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:23.728 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.724+0000 7f4223755700 1 -- 172.21.15.184:0/1281309469 >> 172.21.15.184:0/1281309469 conn(0x7f421c0f0b70 msgr2=0x7f421c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:23.728 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.725+0000 7f4223755700 1 -- 172.21.15.184:0/1281309469 shutdown_connections 2024-02-19T22:20:23.728 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:23.725+0000 7f4223755700 1 -- 172.21.15.184:0/1281309469 wait complete. 2024-02-19T22:20:23.728 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:20:24.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:22.411772+0000 mgr.smithi176.faoogw (mgr.14162) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:24.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:23 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:20:23.724180+0000 mon.smithi176 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.184:0/1281309469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:25.401 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:25.401 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:26.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:25 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:24.411994+0000 mgr.smithi176.faoogw (mgr.14162) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:27.282 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.280+0000 7f043e54b700 1 -- 172.21.15.184:0/3969764060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04380f5410 msgr2=0x7f04380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:27.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.280+0000 7f043e54b700 1 --2- 172.21.15.184:0/3969764060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04380f5410 0x7f04380f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f0428009fb0 tx=0x7f0428009b30).stop 2024-02-19T22:20:27.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.280+0000 7f043e54b700 1 -- 172.21.15.184:0/3969764060 shutdown_connections 2024-02-19T22:20:27.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.280+0000 7f043e54b700 1 --2- 172.21.15.184:0/3969764060 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04380f5410 0x7f04380f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:27.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.280+0000 7f043e54b700 1 -- 172.21.15.184:0/3969764060 >> 172.21.15.184:0/3969764060 conn(0x7f04380f0b70 msgr2=0x7f04380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:27.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.280+0000 7f043e54b700 1 -- 172.21.15.184:0/3969764060 shutdown_connections 2024-02-19T22:20:27.283 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.280+0000 7f043e54b700 1 -- 172.21.15.184:0/3969764060 wait complete. 2024-02-19T22:20:27.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.281+0000 7f043e54b700 1 Processor -- start 2024-02-19T22:20:27.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.281+0000 7f043e54b700 1 -- start start 2024-02-19T22:20:27.285 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.281+0000 7f043e54b700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04380f5410 0x7f04381094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.281+0000 7f043e54b700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0438060280 con 0x7f04380f5410 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.282+0000 7f0437fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04380f5410 0x7f04381094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.282+0000 7f0437fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04380f5410 0x7f04381094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:37246/0 (socket says 172.21.15.184:37246) 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.282+0000 7f0437fff700 1 -- 172.21.15.184:0/607533166 learned_addr learned my addr 172.21.15.184:0/607533166 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.282+0000 7f0437fff700 1 -- 172.21.15.184:0/607533166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0428005040 con 0x7f04380f5410 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.282+0000 7f0437fff700 1 --2- 172.21.15.184:0/607533166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04380f5410 0x7f04381094a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f0428005010 tx=0x7f042800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.282+0000 7f0434ff9700 1 -- 172.21.15.184:0/607533166 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0428009320 con 0x7f04380f5410 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.282+0000 7f043e54b700 1 -- 172.21.15.184:0/607533166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0438060480 con 0x7f04380f5410 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.282+0000 7f0434ff9700 1 -- 172.21.15.184:0/607533166 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0428013380 con 0x7f04380f5410 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.282+0000 7f0434ff9700 1 -- 172.21.15.184:0/607533166 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0428014e50 con 0x7f04380f5410 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.282+0000 7f043e54b700 1 -- 172.21.15.184:0/607533166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0438109bd0 con 0x7f04380f5410 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.283+0000 7f0434ff9700 1 -- 172.21.15.184:0/607533166 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04280095f0 con 0x7f04380f5410 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.283+0000 7f043e54b700 1 -- 172.21.15.184:0/607533166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0438045ad0 con 0x7f04380f5410 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.283+0000 7f0434ff9700 1 --2- 172.21.15.184:0/607533166 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f042402ff10 0x7f04240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:27.286 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.283+0000 7f0434ff9700 1 -- 172.21.15.184:0/607533166 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0428040fa0 con 0x7f04380f5410 2024-02-19T22:20:27.287 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.283+0000 7f04377fe700 1 --2- 172.21.15.184:0/607533166 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f042402ff10 0x7f04240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:27.287 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.284+0000 7f04377fe700 1 --2- 172.21.15.184:0/607533166 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f042402ff10 0x7f04240323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f04200096a0 tx=0x7f0420006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:27.290 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.288+0000 7f0434ff9700 1 -- 172.21.15.184:0/607533166 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0428014420 con 0x7f04380f5410 2024-02-19T22:20:27.498 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.496+0000 7f043e54b700 1 -- 172.21.15.184:0/607533166 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04380fafa0 con 0x7f04380f5410 2024-02-19T22:20:27.499 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.497+0000 7f0434ff9700 1 -- 172.21.15.184:0/607533166 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0428010230 con 0x7f04380f5410 2024-02-19T22:20:27.499 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:20:27.499 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:27.501 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.498+0000 7f043e54b700 1 -- 172.21.15.184:0/607533166 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f042402ff10 msgr2=0x7f04240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:27.501 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.498+0000 7f043e54b700 1 --2- 172.21.15.184:0/607533166 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f042402ff10 0x7f04240323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f04200096a0 tx=0x7f0420006b40).stop 2024-02-19T22:20:27.501 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.498+0000 7f043e54b700 1 -- 172.21.15.184:0/607533166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04380f5410 msgr2=0x7f04381094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:27.501 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.498+0000 7f043e54b700 1 --2- 172.21.15.184:0/607533166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04380f5410 0x7f04381094a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f0428005010 tx=0x7f042800b040).stop 2024-02-19T22:20:27.501 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.498+0000 7f043e54b700 1 -- 172.21.15.184:0/607533166 shutdown_connections 2024-02-19T22:20:27.501 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.498+0000 7f043e54b700 1 --2- 172.21.15.184:0/607533166 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f042402ff10 0x7f04240323d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:27.501 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.499+0000 7f043e54b700 1 --2- 172.21.15.184:0/607533166 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f04380f5410 0x7f04381094a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:27.501 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.499+0000 7f043e54b700 1 -- 172.21.15.184:0/607533166 >> 172.21.15.184:0/607533166 conn(0x7f04380f0b70 msgr2=0x7f04380f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:27.501 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.499+0000 7f043e54b700 1 -- 172.21.15.184:0/607533166 shutdown_connections 2024-02-19T22:20:27.501 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:27.499+0000 7f043e54b700 1 -- 172.21.15.184:0/607533166 wait complete. 2024-02-19T22:20:27.502 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:20:28.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:26.412394+0000 mgr.smithi176.faoogw (mgr.14162) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:28.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:27 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:20:27.498497+0000 mon.smithi176 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.184:0/607533166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:29.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:29.171 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:30.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:29 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:28.412778+0000 mgr.smithi176.faoogw (mgr.14162) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:31.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.027+0000 7f69e8530700 1 -- 172.21.15.184:0/1801024597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69e00f5410 msgr2=0x7f69e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:31.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.027+0000 7f69e8530700 1 --2- 172.21.15.184:0/1801024597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69e00f5410 0x7f69e00f5830 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f69d4009fb0 tx=0x7f69d4009b30).stop 2024-02-19T22:20:31.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.027+0000 7f69e8530700 1 -- 172.21.15.184:0/1801024597 shutdown_connections 2024-02-19T22:20:31.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.027+0000 7f69e8530700 1 --2- 172.21.15.184:0/1801024597 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69e00f5410 0x7f69e00f5830 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:31.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.027+0000 7f69e8530700 1 -- 172.21.15.184:0/1801024597 >> 172.21.15.184:0/1801024597 conn(0x7f69e00f0b70 msgr2=0x7f69e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:31.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.027+0000 7f69e8530700 1 -- 172.21.15.184:0/1801024597 shutdown_connections 2024-02-19T22:20:31.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.027+0000 7f69e8530700 1 -- 172.21.15.184:0/1801024597 wait complete. 2024-02-19T22:20:31.031 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.028+0000 7f69e8530700 1 Processor -- start 2024-02-19T22:20:31.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.028+0000 7f69e8530700 1 -- start start 2024-02-19T22:20:31.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.028+0000 7f69e8530700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69e00f5410 0x7f69e01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:31.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.028+0000 7f69e8530700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e0106cd0 con 0x7f69e00f5410 2024-02-19T22:20:31.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.028+0000 7f69e62cc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69e00f5410 0x7f69e01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:31.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.028+0000 7f69e62cc700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69e00f5410 0x7f69e01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:37274/0 (socket says 172.21.15.184:37274) 2024-02-19T22:20:31.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.028+0000 7f69e62cc700 1 -- 172.21.15.184:0/781110547 learned_addr learned my addr 172.21.15.184:0/781110547 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:20:31.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.029+0000 7f69e62cc700 1 -- 172.21.15.184:0/781110547 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d4005040 con 0x7f69e00f5410 2024-02-19T22:20:31.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.029+0000 7f69e62cc700 1 --2- 172.21.15.184:0/781110547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69e00f5410 0x7f69e01067e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f69d4005010 tx=0x7f69d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:31.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.029+0000 7f69d2ffd700 1 -- 172.21.15.184:0/781110547 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69d4009320 con 0x7f69e00f5410 2024-02-19T22:20:31.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.029+0000 7f69e8530700 1 -- 172.21.15.184:0/781110547 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e0106ed0 con 0x7f69e00f5410 2024-02-19T22:20:31.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.029+0000 7f69d2ffd700 1 -- 172.21.15.184:0/781110547 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69d4013380 con 0x7f69e00f5410 2024-02-19T22:20:31.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.029+0000 7f69d2ffd700 1 -- 172.21.15.184:0/781110547 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69d4014e50 con 0x7f69e00f5410 2024-02-19T22:20:31.032 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.029+0000 7f69e8530700 1 -- 172.21.15.184:0/781110547 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e0107360 con 0x7f69e00f5410 2024-02-19T22:20:31.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.030+0000 7f69d2ffd700 1 -- 172.21.15.184:0/781110547 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69d40095f0 con 0x7f69e00f5410 2024-02-19T22:20:31.033 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.030+0000 7f69e8530700 1 -- 172.21.15.184:0/781110547 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e0045ad0 con 0x7f69e00f5410 2024-02-19T22:20:31.034 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.030+0000 7f69d2ffd700 1 --2- 172.21.15.184:0/781110547 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f69cc02ff10 0x7f69cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:31.034 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.030+0000 7f69d2ffd700 1 -- 172.21.15.184:0/781110547 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69d4040fa0 con 0x7f69e00f5410 2024-02-19T22:20:31.034 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.030+0000 7f69e5acb700 1 --2- 172.21.15.184:0/781110547 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f69cc02ff10 0x7f69cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:31.034 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.030+0000 7f69e5acb700 1 --2- 172.21.15.184:0/781110547 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f69cc02ff10 0x7f69cc0323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f69dc0096a0 tx=0x7f69dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:31.036 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.034+0000 7f69d2ffd700 1 -- 172.21.15.184:0/781110547 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69d4014420 con 0x7f69e00f5410 2024-02-19T22:20:31.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.242+0000 7f69e8530700 1 -- 172.21.15.184:0/781110547 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69e0108210 con 0x7f69e00f5410 2024-02-19T22:20:31.245 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.243+0000 7f69d2ffd700 1 -- 172.21.15.184:0/781110547 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f69d4010230 con 0x7f69e00f5410 2024-02-19T22:20:31.245 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:20:31.245 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:31.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.245+0000 7f69e8530700 1 -- 172.21.15.184:0/781110547 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f69cc02ff10 msgr2=0x7f69cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:31.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.245+0000 7f69e8530700 1 --2- 172.21.15.184:0/781110547 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f69cc02ff10 0x7f69cc0323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f69dc0096a0 tx=0x7f69dc006b40).stop 2024-02-19T22:20:31.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.245+0000 7f69e8530700 1 -- 172.21.15.184:0/781110547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69e00f5410 msgr2=0x7f69e01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:31.247 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.245+0000 7f69e8530700 1 --2- 172.21.15.184:0/781110547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69e00f5410 0x7f69e01067e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f69d4005010 tx=0x7f69d400b040).stop 2024-02-19T22:20:31.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.245+0000 7f69e8530700 1 -- 172.21.15.184:0/781110547 shutdown_connections 2024-02-19T22:20:31.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.245+0000 7f69e8530700 1 --2- 172.21.15.184:0/781110547 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f69cc02ff10 0x7f69cc0323d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:31.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.245+0000 7f69e8530700 1 --2- 172.21.15.184:0/781110547 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69e00f5410 0x7f69e01067e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:31.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.245+0000 7f69e8530700 1 -- 172.21.15.184:0/781110547 >> 172.21.15.184:0/781110547 conn(0x7f69e00f0b70 msgr2=0x7f69e00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:31.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.245+0000 7f69e8530700 1 -- 172.21.15.184:0/781110547 shutdown_connections 2024-02-19T22:20:31.248 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:31.245+0000 7f69e8530700 1 -- 172.21.15.184:0/781110547 wait complete. 2024-02-19T22:20:31.248 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:20:32.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:30.413123+0000 mgr.smithi176.faoogw (mgr.14162) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:32.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:31 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:20:31.244493+0000 mon.smithi176 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.184:0/781110547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:32.913 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:32.914 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:34.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:33 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:32.413494+0000 mgr.smithi176.faoogw (mgr.14162) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:34.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.862+0000 7f4656efd700 1 -- 172.21.15.184:0/1437214702 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46500f5410 msgr2=0x7f46500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:34.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.862+0000 7f4656efd700 1 --2- 172.21.15.184:0/1437214702 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46500f5410 0x7f46500f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f4644009fb0 tx=0x7f4644009b30).stop 2024-02-19T22:20:34.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.862+0000 7f4656efd700 1 -- 172.21.15.184:0/1437214702 shutdown_connections 2024-02-19T22:20:34.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.862+0000 7f4656efd700 1 --2- 172.21.15.184:0/1437214702 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46500f5410 0x7f46500f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:34.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.862+0000 7f4656efd700 1 -- 172.21.15.184:0/1437214702 >> 172.21.15.184:0/1437214702 conn(0x7f46500f0b70 msgr2=0x7f46500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:34.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.862+0000 7f4656efd700 1 -- 172.21.15.184:0/1437214702 shutdown_connections 2024-02-19T22:20:34.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.862+0000 7f4656efd700 1 -- 172.21.15.184:0/1437214702 wait complete. 2024-02-19T22:20:34.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.863+0000 7f4656efd700 1 Processor -- start 2024-02-19T22:20:34.865 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.863+0000 7f4656efd700 1 -- start start 2024-02-19T22:20:34.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.863+0000 7f4656efd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46500f5410 0x7f46500fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:34.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.863+0000 7f4656efd700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46500fb960 con 0x7f46500f5410 2024-02-19T22:20:34.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.863+0000 7f4654c99700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46500f5410 0x7f46500fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:34.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.863+0000 7f4654c99700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46500f5410 0x7f46500fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:37298/0 (socket says 172.21.15.184:37298) 2024-02-19T22:20:34.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.863+0000 7f4654c99700 1 -- 172.21.15.184:0/3509136104 learned_addr learned my addr 172.21.15.184:0/3509136104 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:20:34.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.864+0000 7f4654c99700 1 -- 172.21.15.184:0/3509136104 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4644005040 con 0x7f46500f5410 2024-02-19T22:20:34.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.864+0000 7f4654c99700 1 --2- 172.21.15.184:0/3509136104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46500f5410 0x7f46500fd4e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f46440089f0 tx=0x7f464400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:34.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.864+0000 7f464d7fa700 1 -- 172.21.15.184:0/3509136104 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4644009320 con 0x7f46500f5410 2024-02-19T22:20:34.867 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.864+0000 7f464d7fa700 1 -- 172.21.15.184:0/3509136104 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4644013380 con 0x7f46500f5410 2024-02-19T22:20:34.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.864+0000 7f4656efd700 1 -- 172.21.15.184:0/3509136104 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46500fbb00 con 0x7f46500f5410 2024-02-19T22:20:34.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.864+0000 7f464d7fa700 1 -- 172.21.15.184:0/3509136104 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f464401ed80 con 0x7f46500f5410 2024-02-19T22:20:34.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.864+0000 7f4656efd700 1 -- 172.21.15.184:0/3509136104 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46500fbf90 con 0x7f46500f5410 2024-02-19T22:20:34.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.865+0000 7f464d7fa700 1 -- 172.21.15.184:0/3509136104 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f464401e420 con 0x7f46500f5410 2024-02-19T22:20:34.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.865+0000 7f464d7fa700 1 --2- 172.21.15.184:0/3509136104 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f464002ff60 0x7f4640032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:34.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.865+0000 7f4656efd700 1 -- 172.21.15.184:0/3509136104 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4650059100 con 0x7f46500f5410 2024-02-19T22:20:34.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.865+0000 7f464d7fa700 1 -- 172.21.15.184:0/3509136104 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4644040e70 con 0x7f46500f5410 2024-02-19T22:20:34.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.865+0000 7f464ffff700 1 --2- 172.21.15.184:0/3509136104 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f464002ff60 0x7f4640032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:34.868 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.865+0000 7f464ffff700 1 --2- 172.21.15.184:0/3509136104 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f464002ff60 0x7f4640032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f463c0096a0 tx=0x7f463c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:34.872 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:34.870+0000 7f464d7fa700 1 -- 172.21.15.184:0/3509136104 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46440095f0 con 0x7f46500f5410 2024-02-19T22:20:35.075 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:35.072+0000 7f4656efd700 1 -- 172.21.15.184:0/3509136104 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46500fafa0 con 0x7f46500f5410 2024-02-19T22:20:35.075 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:35.073+0000 7f464d7fa700 1 -- 172.21.15.184:0/3509136104 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f464400e2a0 con 0x7f46500f5410 2024-02-19T22:20:35.075 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:20:35.076 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:35.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:35.075+0000 7f4656efd700 1 -- 172.21.15.184:0/3509136104 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f464002ff60 msgr2=0x7f4640032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:35.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:35.075+0000 7f4656efd700 1 --2- 172.21.15.184:0/3509136104 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f464002ff60 0x7f4640032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f463c0096a0 tx=0x7f463c006b40).stop 2024-02-19T22:20:35.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:35.075+0000 7f4656efd700 1 -- 172.21.15.184:0/3509136104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46500f5410 msgr2=0x7f46500fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:35.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:35.075+0000 7f4656efd700 1 --2- 172.21.15.184:0/3509136104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46500f5410 0x7f46500fd4e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f46440089f0 tx=0x7f464400b040).stop 2024-02-19T22:20:35.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:35.075+0000 7f4656efd700 1 -- 172.21.15.184:0/3509136104 shutdown_connections 2024-02-19T22:20:35.077 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:35.075+0000 7f4656efd700 1 --2- 172.21.15.184:0/3509136104 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f464002ff60 0x7f4640032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:35.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:35.075+0000 7f4656efd700 1 --2- 172.21.15.184:0/3509136104 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46500f5410 0x7f46500fd4e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:35.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:35.075+0000 7f4656efd700 1 -- 172.21.15.184:0/3509136104 >> 172.21.15.184:0/3509136104 conn(0x7f46500f0b70 msgr2=0x7f465005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:35.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:35.075+0000 7f4656efd700 1 -- 172.21.15.184:0/3509136104 shutdown_connections 2024-02-19T22:20:35.078 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:35.075+0000 7f4656efd700 1 -- 172.21.15.184:0/3509136104 wait complete. 2024-02-19T22:20:35.078 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:20:36.135 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:34.413904+0000 mgr.smithi176.faoogw (mgr.14162) 370 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:36.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:35 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:20:35.074573+0000 mon.smithi176 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.184:0/3509136104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:36.800 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:36.800 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:37.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:20:35.978090+0000 mon.smithi176 (mon.0) 338 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:20:37.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:36 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:20:35.978912+0000 mon.smithi176 (mon.0) 339 : audit [INF] from='mgr.14162 172.21.15.176:0/1295315965' entity='mgr.smithi176.faoogw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi176.faoogw/trash_purge_schedule"}]: dispatch 2024-02-19T22:20:38.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:37 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:36.414253+0000 mgr.smithi176.faoogw (mgr.14162) 371 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:38.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.710+0000 7f2990ca2700 1 -- 172.21.15.184:0/2773489813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f298c0f62a0 msgr2=0x7f298c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:38.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.710+0000 7f2990ca2700 1 --2- 172.21.15.184:0/2773489813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f298c0f62a0 0x7f298c0f66c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f297c009fb0 tx=0x7f297c009b30).stop 2024-02-19T22:20:38.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.710+0000 7f2990ca2700 1 -- 172.21.15.184:0/2773489813 shutdown_connections 2024-02-19T22:20:38.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.710+0000 7f2990ca2700 1 --2- 172.21.15.184:0/2773489813 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f298c0f62a0 0x7f298c0f66c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:38.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.710+0000 7f2990ca2700 1 -- 172.21.15.184:0/2773489813 >> 172.21.15.184:0/2773489813 conn(0x7f298c0f1aa0 msgr2=0x7f298c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:38.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.710+0000 7f2990ca2700 1 -- 172.21.15.184:0/2773489813 shutdown_connections 2024-02-19T22:20:38.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.710+0000 7f2990ca2700 1 -- 172.21.15.184:0/2773489813 wait complete. 2024-02-19T22:20:38.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.711+0000 7f2990ca2700 1 Processor -- start 2024-02-19T22:20:38.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.711+0000 7f2990ca2700 1 -- start start 2024-02-19T22:20:38.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.711+0000 7f2990ca2700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f298c0f62a0 0x7f298c1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:38.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.711+0000 7f2990ca2700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c106ac0 con 0x7f298c0f62a0 2024-02-19T22:20:38.714 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.711+0000 7f298b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f298c0f62a0 0x7f298c1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:38.715 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.711+0000 7f298b7fe700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f298c0f62a0 0x7f298c1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:48240/0 (socket says 172.21.15.184:48240) 2024-02-19T22:20:38.715 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.711+0000 7f298b7fe700 1 -- 172.21.15.184:0/2529806487 learned_addr learned my addr 172.21.15.184:0/2529806487 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:20:38.715 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.712+0000 7f298b7fe700 1 -- 172.21.15.184:0/2529806487 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297c005040 con 0x7f298c0f62a0 2024-02-19T22:20:38.715 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.712+0000 7f298b7fe700 1 --2- 172.21.15.184:0/2529806487 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f298c0f62a0 0x7f298c1065d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f297c0089f0 tx=0x7f297c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:38.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.712+0000 7f2973fff700 1 -- 172.21.15.184:0/2529806487 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f297c009320 con 0x7f298c0f62a0 2024-02-19T22:20:38.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.712+0000 7f2990ca2700 1 -- 172.21.15.184:0/2529806487 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298c106cc0 con 0x7f298c0f62a0 2024-02-19T22:20:38.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.712+0000 7f2973fff700 1 -- 172.21.15.184:0/2529806487 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f297c013380 con 0x7f298c0f62a0 2024-02-19T22:20:38.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.712+0000 7f2973fff700 1 -- 172.21.15.184:0/2529806487 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f297c014e20 con 0x7f298c0f62a0 2024-02-19T22:20:38.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.712+0000 7f2990ca2700 1 -- 172.21.15.184:0/2529806487 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298c107150 con 0x7f298c0f62a0 2024-02-19T22:20:38.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.713+0000 7f2990ca2700 1 -- 172.21.15.184:0/2529806487 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f298c045ad0 con 0x7f298c0f62a0 2024-02-19T22:20:38.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.713+0000 7f2973fff700 1 -- 172.21.15.184:0/2529806487 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f297c0095f0 con 0x7f298c0f62a0 2024-02-19T22:20:38.716 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.713+0000 7f2973fff700 1 --2- 172.21.15.184:0/2529806487 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f297402ff10 0x7f29740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:38.717 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.713+0000 7f2973fff700 1 -- 172.21.15.184:0/2529806487 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f297c040fb0 con 0x7f298c0f62a0 2024-02-19T22:20:38.717 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.713+0000 7f298affd700 1 --2- 172.21.15.184:0/2529806487 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f297402ff10 0x7f29740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:38.717 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.714+0000 7f298affd700 1 --2- 172.21.15.184:0/2529806487 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f297402ff10 0x7f29740323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f29800096a0 tx=0x7f2980006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:38.720 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.718+0000 7f2973fff700 1 -- 172.21.15.184:0/2529806487 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f297c06d2b0 con 0x7f298c0f62a0 2024-02-19T22:20:38.924 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.922+0000 7f2990ca2700 1 -- 172.21.15.184:0/2529806487 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f298c107fa0 con 0x7f298c0f62a0 2024-02-19T22:20:38.924 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.922+0000 7f2973fff700 1 -- 172.21.15.184:0/2529806487 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f297c014420 con 0x7f298c0f62a0 2024-02-19T22:20:38.924 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:20:38.925 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:38.926 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.924+0000 7f2990ca2700 1 -- 172.21.15.184:0/2529806487 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f297402ff10 msgr2=0x7f29740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:38.926 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.924+0000 7f2990ca2700 1 --2- 172.21.15.184:0/2529806487 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f297402ff10 0x7f29740323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f29800096a0 tx=0x7f2980006b40).stop 2024-02-19T22:20:38.926 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.924+0000 7f2990ca2700 1 -- 172.21.15.184:0/2529806487 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f298c0f62a0 msgr2=0x7f298c1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:38.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.924+0000 7f2990ca2700 1 --2- 172.21.15.184:0/2529806487 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f298c0f62a0 0x7f298c1065d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f297c0089f0 tx=0x7f297c00b040).stop 2024-02-19T22:20:38.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.924+0000 7f2990ca2700 1 -- 172.21.15.184:0/2529806487 shutdown_connections 2024-02-19T22:20:38.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.924+0000 7f2990ca2700 1 --2- 172.21.15.184:0/2529806487 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7f297402ff10 0x7f29740323d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:38.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.924+0000 7f2990ca2700 1 --2- 172.21.15.184:0/2529806487 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f298c0f62a0 0x7f298c1065d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:38.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.924+0000 7f2990ca2700 1 -- 172.21.15.184:0/2529806487 >> 172.21.15.184:0/2529806487 conn(0x7f298c0f1aa0 msgr2=0x7f298c0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:38.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.924+0000 7f2990ca2700 1 -- 172.21.15.184:0/2529806487 shutdown_connections 2024-02-19T22:20:38.927 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:38.924+0000 7f2990ca2700 1 -- 172.21.15.184:0/2529806487 wait complete. 2024-02-19T22:20:38.927 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:20:40.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:38.414575+0000 mgr.smithi176.faoogw (mgr.14162) 372 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:40.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:39 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:20:38.923831+0000 mon.smithi176 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.184:0/2529806487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:40.657 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:40.657 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:42.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:41 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: cluster 2024-02-19T22:20:40.414939+0000 mgr.smithi176.faoogw (mgr.14162) 373 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:42.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.521+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2004260666 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2e40f62a0 msgr2=0x7fa2e40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:42.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.521+0000 7fa2e3fff700 1 --2- 172.21.15.184:0/2004260666 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2e40f62a0 0x7fa2e40f66c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fa2d8009fb0 tx=0x7fa2d8009b30).stop 2024-02-19T22:20:42.523 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.521+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2004260666 shutdown_connections 2024-02-19T22:20:42.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.521+0000 7fa2e3fff700 1 --2- 172.21.15.184:0/2004260666 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2e40f62a0 0x7fa2e40f66c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:42.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.521+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2004260666 >> 172.21.15.184:0/2004260666 conn(0x7fa2e40f1aa0 msgr2=0x7fa2e40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:42.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.521+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2004260666 shutdown_connections 2024-02-19T22:20:42.524 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.521+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2004260666 wait complete. 2024-02-19T22:20:42.525 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.522+0000 7fa2e3fff700 1 Processor -- start 2024-02-19T22:20:42.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.522+0000 7fa2e3fff700 1 -- start start 2024-02-19T22:20:42.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.522+0000 7fa2e3fff700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2e40f62a0 0x7fa2e41087a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:42.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.522+0000 7fa2e3fff700 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e4108c90 con 0x7fa2e40f62a0 2024-02-19T22:20:42.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.522+0000 7fa2e2ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2e40f62a0 0x7fa2e41087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:42.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.522+0000 7fa2e2ffd700 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2e40f62a0 0x7fa2e41087a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.184:48244/0 (socket says 172.21.15.184:48244) 2024-02-19T22:20:42.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.522+0000 7fa2e2ffd700 1 -- 172.21.15.184:0/2113345831 learned_addr learned my addr 172.21.15.184:0/2113345831 (peer_addr_for_me v2:172.21.15.184:0/0) 2024-02-19T22:20:42.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.522+0000 7fa2e2ffd700 1 -- 172.21.15.184:0/2113345831 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d8005040 con 0x7fa2e40f62a0 2024-02-19T22:20:42.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.522+0000 7fa2e2ffd700 1 --2- 172.21.15.184:0/2113345831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2e40f62a0 0x7fa2e41087a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fa2d80089f0 tx=0x7fa2d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:42.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.522+0000 7fa2cb7fe700 1 -- 172.21.15.184:0/2113345831 <== mon.0 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2d8009320 con 0x7fa2e40f62a0 2024-02-19T22:20:42.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.522+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2113345831 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e4108e90 con 0x7fa2e40f62a0 2024-02-19T22:20:42.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.522+0000 7fa2cb7fe700 1 -- 172.21.15.184:0/2113345831 <== mon.0 v2:172.21.15.176:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2d8013380 con 0x7fa2e40f62a0 2024-02-19T22:20:42.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.522+0000 7fa2cb7fe700 1 -- 172.21.15.184:0/2113345831 <== mon.0 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2d8014e20 con 0x7fa2e40f62a0 2024-02-19T22:20:42.526 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.523+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2113345831 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e4109320 con 0x7fa2e40f62a0 2024-02-19T22:20:42.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.523+0000 7fa2cb7fe700 1 -- 172.21.15.184:0/2113345831 <== mon.0 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2d8014420 con 0x7fa2e40f62a0 2024-02-19T22:20:42.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.523+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2113345831 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e4045ad0 con 0x7fa2e40f62a0 2024-02-19T22:20:42.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.523+0000 7fa2cb7fe700 1 --2- 172.21.15.184:0/2113345831 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa2cc02ff10 0x7fa2cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:42.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.523+0000 7fa2cb7fe700 1 -- 172.21.15.184:0/2113345831 <== mon.0 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2d8040ff0 con 0x7fa2e40f62a0 2024-02-19T22:20:42.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.524+0000 7fa2e27fc700 1 --2- 172.21.15.184:0/2113345831 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa2cc02ff10 0x7fa2cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:42.527 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.524+0000 7fa2e27fc700 1 --2- 172.21.15.184:0/2113345831 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa2cc02ff10 0x7fa2cc0323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fa2d40096a0 tx=0x7fa2d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:42.530 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.528+0000 7fa2cb7fe700 1 -- 172.21.15.184:0/2113345831 <== mon.0 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2d80095f0 con 0x7fa2e40f62a0 2024-02-19T22:20:42.737 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.735+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2113345831 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2e406b320 con 0x7fa2e40f62a0 2024-02-19T22:20:42.738 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.736+0000 7fa2cb7fe700 1 -- 172.21.15.184:0/2113345831 <== mon.0 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa2d801db70 con 0x7fa2e40f62a0 2024-02-19T22:20:42.738 INFO:teuthology.orchestra.run.smithi184.stdout: 2024-02-19T22:20:42.738 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":1,"fsid":"36a8f1ec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:07:31.982550Z","created":"2024-02-19T22:07:31.982550Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi176","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:42.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.737+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2113345831 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa2cc02ff10 msgr2=0x7fa2cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:42.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.737+0000 7fa2e3fff700 1 --2- 172.21.15.184:0/2113345831 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa2cc02ff10 0x7fa2cc0323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fa2d40096a0 tx=0x7fa2d4006b40).stop 2024-02-19T22:20:42.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.737+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2113345831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2e40f62a0 msgr2=0x7fa2e41087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:42.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.737+0000 7fa2e3fff700 1 --2- 172.21.15.184:0/2113345831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2e40f62a0 0x7fa2e41087a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fa2d80089f0 tx=0x7fa2d800b040).stop 2024-02-19T22:20:42.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.737+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2113345831 shutdown_connections 2024-02-19T22:20:42.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.737+0000 7fa2e3fff700 1 --2- 172.21.15.184:0/2113345831 >> [v2:172.21.15.176:6800/3746885080,v1:172.21.15.176:6801/3746885080] conn(0x7fa2cc02ff10 0x7fa2cc0323d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:42.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.737+0000 7fa2e3fff700 1 --2- 172.21.15.184:0/2113345831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa2e40f62a0 0x7fa2e41087a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:42.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.737+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2113345831 >> 172.21.15.184:0/2113345831 conn(0x7fa2e40f1aa0 msgr2=0x7fa2e40f3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:42.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.737+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2113345831 shutdown_connections 2024-02-19T22:20:42.740 INFO:teuthology.orchestra.run.smithi184.stderr:2024-02-19T22:20:42.738+0000 7fa2e3fff700 1 -- 172.21.15.184:0/2113345831 wait complete. 2024-02-19T22:20:42.741 INFO:teuthology.orchestra.run.smithi184.stderr:dumped monmap epoch 1 2024-02-19T22:20:43.136 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:42 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: audit 2024-02-19T22:20:42.737004+0000 mon.smithi176 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.184:0/2113345831' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:43.466 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-19T22:20:43.468 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-19T22:20:43.468 DEBUG:teuthology.orchestra.run.smithi176:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:20:43.493 DEBUG:teuthology.orchestra.run.smithi184:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:20:43.512 INFO:tasks.cephadm:Stopping all daemons... 2024-02-19T22:20:43.512 INFO:tasks.cephadm.mon.smithi176:Stopping mon.smithi176... 2024-02-19T22:20:43.512 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi176 2024-02-19T22:20:43.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:43 smithi176 systemd[1]: Stopping Ceph mon.smithi176 for 36a8f1ec-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:20:43.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:20:43.680+0000 7fc496d84700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi176 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T22:20:43.886 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:43 smithi176 ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176[38110]: debug 2024-02-19T22:20:43.680+0000 7fc496d84700 -1 mon.smithi176@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T22:20:44.285 INFO:journalctl@ceph.mon.smithi176.smithi176.stdout:Feb 19 22:20:44 smithi176 podman[44009]: ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715-mon.smithi176 2024-02-19T22:20:44.288 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi176.service' 2024-02-19T22:20:44.335 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T22:20:44.335 INFO:tasks.cephadm.mon.smithi176:Stopped mon.smithi176 2024-02-19T22:20:44.335 INFO:tasks.cephadm.mon.smithi184:Stopping mon.smithi184... 2024-02-19T22:20:44.336 DEBUG:teuthology.orchestra.run.smithi184:> sudo systemctl stop ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi184 2024-02-19T22:20:44.374 INFO:teuthology.orchestra.run.smithi184.stderr:Failed to stop ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi184.service: Unit ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi184.service not loaded. 2024-02-19T22:20:44.375 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-19T22:20:44.376 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi184" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi184 with status 5: 'sudo systemctl stop ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi184' 2024-02-19T22:20:44.377 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-19T22:20:44.379 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566805/remote/ubuntu@smithi176.front.sepia.ceph.com/crash 2024-02-19T22:20:44.380 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/crash -- . 2024-02-19T22:20:44.421 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566805/remote/ubuntu@smithi184.front.sepia.ceph.com/crash 2024-02-19T22:20:44.422 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/crash -- . 2024-02-19T22:20:44.457 INFO:teuthology.orchestra.run.smithi184.stderr:tar: /var/lib/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/crash: Cannot open: No such file or directory 2024-02-19T22:20:44.458 INFO:teuthology.orchestra.run.smithi184.stderr:tar: Error is not recoverable: exiting now 2024-02-19T22:20:44.459 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-19T22:20:44.459 DEBUG:teuthology.orchestra.run.smithi176:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v 'Degraded data redundancy' | egrep -v '\(PG_' | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'deprecated feature inline_data' | egrep -v 'compat changed unexpectedly' | egrep -v 'overall HEALTH_' | egrep -v OSD_DOWN | egrep -v OSD_ | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-19T22:20:44.494 INFO:tasks.cephadm:Compressing logs... 2024-02-19T22:20:44.494 DEBUG:teuthology.orchestra.run.smithi176:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T22:20:44.536 DEBUG:teuthology.orchestra.run.smithi184:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T22:20:44.561 INFO:teuthology.orchestra.run.smithi176.stderr:find: ‘/var/log/rbd-target-api’gzip : No such file or directory-5 2024-02-19T22:20:44.561 INFO:teuthology.orchestra.run.smithi176.stderr:--verbose -- /var/log/ceph/cephadm.log 2024-02-19T22:20:44.561 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph-mgr.smithi176.faoogw.log 2024-02-19T22:20:44.562 INFO:teuthology.orchestra.run.smithi176.stderr:gzip /var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T22:20:44.562 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph-mgr.smithi176.faoogw.log: gzip -5 --verbose -- /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph.audit.log 2024-02-19T22:20:44.563 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph.cephadm.log 2024-02-19T22:20:44.563 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph-mon.smithi176.log 2024-02-19T22:20:44.563 INFO:teuthology.orchestra.run.smithi176.stderr: 88.4% -- replaced with /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T22:20:44.563 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph.log 2024-02-19T22:20:44.564 INFO:teuthology.orchestra.run.smithi176.stderr: 90.3% -- replaced with /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph.cephadm.log.gz 2024-02-19T22:20:44.564 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph-mon.smithi176.log: 88.8% -- replaced with /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph.audit.log.gz 2024-02-19T22:20:44.564 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph.log: 88.7% -- replaced with /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph.log.gz 2024-02-19T22:20:44.567 INFO:teuthology.orchestra.run.smithi184.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-19T22:20:44.567 INFO:teuthology.orchestra.run.smithi184.stderr:: No such file or directory 2024-02-19T22:20:44.567 INFO:teuthology.orchestra.run.smithi184.stderr:gzip -5 --verbose -- /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T22:20:44.567 INFO:teuthology.orchestra.run.smithi176.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T22:20:44.568 INFO:teuthology.orchestra.run.smithi184.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph-volume.log: 88.5% -- replaced with /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T22:20:44.569 INFO:teuthology.orchestra.run.smithi184.stderr: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T22:20:44.570 INFO:teuthology.orchestra.run.smithi184.stderr: 2024-02-19T22:20:44.570 INFO:teuthology.orchestra.run.smithi184.stderr:real 0m0.015s 2024-02-19T22:20:44.570 INFO:teuthology.orchestra.run.smithi184.stderr:user 0m0.012s 2024-02-19T22:20:44.570 INFO:teuthology.orchestra.run.smithi184.stderr:sys 0m0.013s 2024-02-19T22:20:44.659 INFO:teuthology.orchestra.run.smithi176.stderr: 92.7% -- replaced with /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph-mgr.smithi176.faoogw.log.gz 2024-02-19T22:20:44.734 INFO:teuthology.orchestra.run.smithi176.stderr: 90.6% -- replaced with /var/log/ceph/36a8f1ec-cf73-11ee-95bb-87774f69a715/ceph-mon.smithi176.log.gz 2024-02-19T22:20:44.736 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-02-19T22:20:44.736 INFO:teuthology.orchestra.run.smithi176.stderr:real 0m0.184s 2024-02-19T22:20:44.736 INFO:teuthology.orchestra.run.smithi176.stderr:user 0m0.280s 2024-02-19T22:20:44.736 INFO:teuthology.orchestra.run.smithi176.stderr:sys 0m0.019s 2024-02-19T22:20:44.736 INFO:tasks.cephadm:Archiving logs... 2024-02-19T22:20:44.736 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566805/remote/ubuntu@smithi176.front.sepia.ceph.com/log 2024-02-19T22:20:44.738 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-19T22:20:44.855 DEBUG:teuthology.misc:Transferring archived files from smithi184:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566805/remote/ubuntu@smithi184.front.sepia.ceph.com/log 2024-02-19T22:20:44.856 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-19T22:20:44.891 INFO:tasks.cephadm:Removing cluster... 2024-02-19T22:20:44.891 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 --force 2024-02-19T22:20:48.020 DEBUG:teuthology.orchestra.run.smithi184:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36a8f1ec-cf73-11ee-95bb-87774f69a715 --force 2024-02-19T22:23:36.969 INFO:tasks.cephadm:Removing cephadm ... 2024-02-19T22:23:36.970 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-19T22:23:36.994 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-19T22:23:37.013 INFO:tasks.cephadm:Teardown complete 2024-02-19T22:23:37.013 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi184 with status 5: 'sudo systemctl stop ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi184' 2024-02-19T22:23:37.229 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=30fdf58843604ea28ecb59d6ffb8f8cb Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi184 with status 5: 'sudo systemctl stop ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi184' 2024-02-19T22:23:37.233 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-19T22:23:37.243 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-19T22:23:37.271 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-19T22:23:37.271 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T22:23:37.296 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T22:23:37.396 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-19T22:23:37.405 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-19T22:23:37.405 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:23:37.408 DEBUG:teuthology.orchestra.run.smithi184:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:23:37.424 INFO:teuthology.orchestra.run.smithi176.stderr:bash: ntpq: command not found 2024-02-19T22:23:37.424 INFO:teuthology.orchestra.run.smithi184.stderr:bash: ntpq: command not found 2024-02-19T22:23:37.428 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:23:37.428 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-02-19T22:23:37.428 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 +1823us[+1823us] +/- 102ms 2024-02-19T22:23:37.428 INFO:teuthology.orchestra.run.smithi176.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -1547us[-1564us] +/- 54ms 2024-02-19T22:23:37.428 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +905us[ +905us] +/- 88ms 2024-02-19T22:23:37.428 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:23:37.430 INFO:teuthology.orchestra.run.smithi184.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:23:37.430 INFO:teuthology.orchestra.run.smithi184.stdout:=============================================================================== 2024-02-19T22:23:37.430 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 +1818us[+1829us] +/- 102ms 2024-02-19T22:23:37.430 INFO:teuthology.orchestra.run.smithi184.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 -1578us[-1566us] +/- 54ms 2024-02-19T22:23:37.430 INFO:teuthology.orchestra.run.smithi184.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +889us[ +901us] +/- 88ms 2024-02-19T22:23:37.430 INFO:teuthology.orchestra.run.smithi184.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:23:37.431 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-19T22:23:37.440 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-19T22:23:37.441 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-19T22:23:37.480 DEBUG:teuthology.orchestra.run.smithi176:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T22:23:37.543 DEBUG:teuthology.orchestra.run.smithi184:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T22:23:37.606 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T22:23:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:23:37.635 DEBUG:teuthology.orchestra.run.smithi184:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T22:23:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:23:37.664 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-19T22:23:37.674 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-19T22:23:37.698 INFO:teuthology.task.internal:Duration was 1574.839374 seconds 2024-02-19T22:23:37.699 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-19T22:23:37.746 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-19T22:23:37.747 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T22:23:37.749 DEBUG:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T22:23:37.792 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T22:23:37.795 INFO:teuthology.orchestra.run.smithi184.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T22:23:37.896 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-19T22:23:37.896 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2024-02-19T22:23:37.896 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T22:23:37.918 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi184.front.sepia.ceph.com 2024-02-19T22:23:37.918 DEBUG:teuthology.orchestra.run.smithi184:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T22:23:37.940 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-19T22:23:37.940 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T22:23:37.961 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T22:23:38.029 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-19T22:23:38.029 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T22:23:38.031 DEBUG:teuthology.orchestra.run.smithi184:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T22:23:38.144 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-19T22:23:38.155 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-19T22:23:38.155 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T22:23:38.182 DEBUG:teuthology.orchestra.run.smithi184:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T22:23:38.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-19T22:23:38.218 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:23:38.225 DEBUG:teuthology.orchestra.run.smithi184:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:23:38.249 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2024-02-19T22:23:38.275 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = core 2024-02-19T22:23:38.307 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:23:38.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:23:38.325 DEBUG:teuthology.orchestra.run.smithi184:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:23:38.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:23:38.364 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-19T22:23:38.375 INFO:teuthology.task.internal:Transferring archived files... 2024-02-19T22:23:38.375 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566805/remote/smithi176 2024-02-19T22:23:38.376 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T22:23:38.415 DEBUG:teuthology.misc:Transferring archived files from smithi184:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566805/remote/smithi184 2024-02-19T22:23:38.415 DEBUG:teuthology.orchestra.run.smithi184:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T22:23:38.454 INFO:teuthology.task.internal:Removing archive directory... 2024-02-19T22:23:38.454 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T22:23:38.456 DEBUG:teuthology.orchestra.run.smithi184:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T22:23:38.506 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-19T22:23:38.517 INFO:teuthology.task.internal:Not uploading archives. 2024-02-19T22:23:38.517 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-19T22:23:38.539 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-19T22:23:38.539 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T22:23:38.542 DEBUG:teuthology.orchestra.run.smithi184:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T22:23:38.557 INFO:teuthology.orchestra.run.smithi176.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 22:23 /home/ubuntu/cephtest 2024-02-19T22:23:38.563 INFO:teuthology.orchestra.run.smithi184.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 22:23 /home/ubuntu/cephtest 2024-02-19T22:23:38.565 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-19T22:23:38.590 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-19T22:23:38.690 INFO:teuthology.nuke:Checking targets against current locks 2024-02-19T22:23:38.725 DEBUG:teuthology.nuke:shortname: smithi176 2024-02-19T22:23:38.725 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T22:23:38.770 DEBUG:teuthology.nuke:shortname: smithi184 2024-02-19T22:23:38.771 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T22:23:38.781 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566805', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:49:15.896682', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPf///913xOrL2NMpXOzOI/M8zYvLGCOT37fQKf5ikaK68NoO9PiHo6l+sRoZmZeWo5+OdjegIenv9V9zakyIkg='} 2024-02-19T22:23:38.827 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi184.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566805', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:49:15.898544', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNiXeisfqaKrCGXPH/j/m8EDYZT0uEObYnnTxCi4UKUJ1XvQI2qhHoJCoEJkLXSVTJW8IMDsRnpa9A1qDfh7wgg='} 2024-02-19T22:23:38.867 INFO:teuthology.orchestra.console.smithi176:Power off 2024-02-19T22:23:38.867 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T22:23:38.913 INFO:teuthology.orchestra.console.smithi184:Power off 2024-02-19T22:23:38.913 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T22:23:38.923 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2024-02-19T22:23:38.923 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:23:38.966 DEBUG:teuthology.orchestra.console.smithi184:power off output: Chassis Power Control: Down/Off 2024-02-19T22:23:38.966 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:23:38.974 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-02-19T22:23:39.014 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-02-19T22:23:42.975 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:23:43.015 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:23:43.086 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-02-19T22:23:43.124 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is on 2024-02-19T22:23:47.091 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:23:47.125 DEBUG:teuthology.orchestra.console.smithi184:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:23:47.202 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-02-19T22:23:47.234 DEBUG:teuthology.orchestra.console.smithi184:check power output: Chassis Power is off 2024-02-19T22:23:47.304 INFO:teuthology.orchestra.console.smithi176:Power off completed 2024-02-19T22:23:47.335 INFO:teuthology.orchestra.console.smithi184:Power off completed 2024-02-19T22:23:47.440 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1574.8393740653992 failure_reason: 'Command failed on smithi184 with status 5: ''sudo systemctl stop ceph-36a8f1ec-cf73-11ee-95bb-87774f69a715@mon.smithi184''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=30fdf58843604ea28ecb59d6ffb8f8cb status: fail success: false 2024-02-19T22:23:47.440 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T22:23:47.515 INFO:teuthology.run:FAIL